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PREFACE 


Virtually  since  its  beginning  as  a  research  center,  Rand  has  been 
examining  the  process  of  developing  and  buying  major  weapon  systems. 

Studies  have  extended  over  a  range  of  topics  from  consideration  of  the 
effectiveness  of  different  types  of  contracts  to  an  evaluation  of  pro¬ 
cedures  appropriate  to  the  acquisition  of  one  small  set  of  components. 

The  acquisition  of  major  weapon  systems  has  become  of  particular  concern 
lately  because  new  weapons  reaching  the  operational  inventory  often 
cost  substantially  more  than  was  initially  proposed.  That  is  not  a 
novel  phenomenon,  of  course,  although  it  is  receiving  renewed  attention 
at  present. 

the  spring  of  1969,  the  Deputy  Chief  of  Staff,  Research  and 
Development,  for  the  United  States  Air  Force,  and  the  Director  of 
Defense  Research  and  Engineering  (DDR&E)  approached  Rand  almost  con¬ 
currently  with  a  set  of  questions  that  arose  from  increasing  concern 
about  this  cost  growth  syndrome.  The  Air  Force  was  interested  in  the 
possibility  of  improving  the  accuracy  of  the  cost  estimating  process 
that  underlay  early  proposals.  DDR&E  wanted  to  know  if  it  was  possible 
to  distinguish  between  the  program  management  effectiveness  of  the  1950s 
and  the  1960s  and  to  identify  those  elements  of  acquisition  policy  that 
were  responsible  for  any  important  differences.  Within  Rand,  a  set  of 
intensive  studies  of  the  weapon  system  acquisition  process  grew  out  of 
these  inquiries  and  from  the  base  of  earlier  research.  This  report  is 
a  review  and  summary  of  some  of  the  research  and  findings  of  the  Rand 
system  acquisition  study  in  1969  and  1970  and  a  statement  of  some  policy 
implications.  Much  of  the  detailed  work  performed  by  various  members 
of  this  group  has  been  separately  reported  to  both  DDR&E  and  the  Air 
Force.  Three  reports  had  been  published  by  1971,  and  several  others 
are  in  preparation.  However,  owing  to  the  rapid  pace  of  the  work  and 

*r.  L.  Perry,  D.  DiSalvo,  G.  R.  Hall,  A.  J.  Harman,  G.  S.  Levenson, 

G.  K.  Smith  and  J.  P.  Stucker,  System  Acquisition  Experience ,  RM-6072-PR, 
November  1969;  Alvin  J.  Harman,  assisted  by  Susan  Henri chsen,  A  Method¬ 
ology  for  Cost  Factor  Comparison  and  Prediction s  RM-6269-ARPA ,  August 
1970;  Arthur  J.  Alexander,  R&D  in  Soviet  Aviation 3  R-589-PR,  November  1970. 
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the  relatively  high  interest  of  both  the  Air  Force  and  DDR&E  in  being 
advised  even  of  preliminary  and  interim  findings,  much  of  the  research 
has  been  reported  in  briefings.  This  report  is  largely  concerned  with 
work  not  previously  described  in  written  form.  Much  of  the  research 
reviewed  here  will  be  covered  in  greater  detail  in  reports  now  being 
prepared. 

This  report  should  be  of  particular  interest  to  the  various  DoD, 
USAF,  industry,  and  research-institute  groups  concerned  with  possible 
changes  in  present  system  acquisition  procedures,  processes,  and  insti 


tutions . 
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SUMMARY 


Notwithstanding  determined  efforts  during  the  1960s  to  improve 
the  outcome  of  major  system  acquisition  programs  by  altering  contractual 
approaches  and  by  introducing  a  variety  of  management  reforms,  typical 
programs  continued  to  exhibit  an  average  cost  growth  of  about  40  per¬ 
cent  (corrected  for  quantity  changes  and  inflation) ,  a  schedule  slip 
of  about  15  percent,  and  final  system  performance  that  was  likely  to 
deviate  by  30  or  40  percent  from  the  original  specification.  Such 
findings  have  been  reported  in  earlier  studies.  During  the  present 
study,  an  examination  of  the  basic  factors  contributing  to  cost  growth 
suggested  two  avenues  for  improvement: 

1.  The  cost  estimation  process  could  be  improved  so  that 
cost  predictions  made  early  in  a  program  would  more 
nearly  correspond  to  the  cost  actually  experienced. 

2.  More  fundamental  improvements  seem  likely  to  be  achieved 
through  some  basic  changes  in  the  acquisition  process. 

A  potentially  important  step  in  cost  estimation  improvement  would 
be  to  incorporate  explicitly  in  the  estimating  relationship  a  measure 
of  the  technical  advance  sought  in  the  program,  because  an  analysis 
of  past  programs  suggests  a  strong  correlation  between  the  degree  of 
advance  sought  and  the  subsequent  cost  growth.  However,  cost  estimating 
improvements  of  this  type  serve  only  to  reduce  the  unpleasant  surprises 
caused  by  unanticipated  cost  growth  and  do  little  to  improve  the  acqui¬ 
sition  process  itself. 

Observations  of  the  outcomes  of  some  recent  European  weapon  system 
programs  and  some  U.S.  weapon  system  programs  that  were  conducted  out¬ 
side  the  normal  DoD  acquisition  channels  suggest  that  two  more  basic 
changes  are  worth  considering.  Both  represent  an  attempt  to  cope  with 
the  substantial  uncertainties  and  inevitable  revisions  that  occur  in 
military  hardware  programs  after  they  have  begun  development.  The 
first  change  is  to  separate  the  development  phase  from  the  subsequent 
production  phase,  both  sequentially  and  contractually.  Such  an  adjust¬ 
ment  should  appreciably  reduce  the  incidence  of  very  expensive 


-vi- 


production  line  changes  that  are  often  caused  by  technical  problems 
discovered  late  in  the  development  program.  The  available  evidence 
indicates  that  a  sequential,  incremental  acquisition  strategy  would 
not  appreciably  increase  the  time  needed  to  progress  from  development 
start  to  the  first  true  operational  capability.  The  second  suggested 
change  is  to  conduct  the  initial  portion  of  the  development  phase  in 
a  highly  austere  manner,  concentrating  first  on  demonstrating  system 
performance  while  deferring  the  more  expensive  tasks  of  detailed  pro¬ 
duction  design  and  demonstration  of  reliability.  This  early  emphasis 
on  performance  demonstration  should  reveal  most  of  the  technical  prob¬ 
lems  at  a  fraction  of  the  cost  characteristic  of  current  programs. 
Furthermore,  initial  austerity  should  in  many  instances  permit  multiple 
source  development,  if  that  is  otherwise  desirable,  so  that  a  truly 
competitive  stance  could  be  maintained  in  contract  negotiations  much 
further  into  a  program  than  is  now  the  case. 

Finally,  it  seems  clear  that  any  system  acquisition  policy  should 
be  flexible.  The  changes  outlined  above  should  be  applied  to  individual 
programs  as  individually  appropriate.  However,  the  evidence  suggests 
that  the  normal  mode  of  acquiring  weapon  systems  during  the  1970s 
probably  should  be  based  on  an  incremental  acquisition  strategy,  with 
the  exceptions  being  determined  by  such  special  considerations  as 
might  occur. 
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I.  THE  WEAPONS  ACQUISITION  SCENE 


The  original  objective  of  the  Rand  system  acquisition  study  was 

A 

to  determine  the  true  extent  of  program  growth  in  major  Department 
of  Defense  weapon  systems  during  recent  years  and  to  establish,  if 
possible,  the  causes  and  indicators  of  differences  between  predicted 
and  actual  outcomes  of  major  programs.  Subsequently,  an  effort  was 
made  to  propose  improved  procedures  for  estimating  the  outcomes  of 
programs;  although  this  effort  encompassed  the  refinement  of  classic 
cost  estimating  procedures,  it  was  not  limited  to  that  approach. 

Finally,  as  the  work  proceeded  it  became  apparent  that  even  on  the 
strength  of  preliminary  findings  it  was  pos'sible  to  propose  changes  in 
present  acquisition  procedures  that  seemed  likely  to  improve  the 
control  of  the  system  acquisition  process. 

The  initial  step  in  the  study  was  to  develop  a  detailed  and  reli¬ 
able  set  of  historical  data  on  recent  development  programs  so  that  sub¬ 
sequent  analysis  would  be  based  on  documentable  facts  rather  than  on 
casual  or  subjective  observations.  The  request  from  DDR&E  to  compare 
programs  of  the  1960s  decade  with  those  of  the  1950s,  in  terms  of  how 
well  program  cost,  schedule,  and  performance  outcomes  compared  with 
predictions,  made  it  possible  to  secure  data  from  all  three  services. 
With  DDR&E  assistance,  Rand  researchers  obtained  basic  information  on 
21  major  weapon  systems  developed  during  the  1960s,  representing  acqui¬ 
sitions  of  each  of  the  three  services.  The  systems  originally  proposed 
for  inclusion  were  selected  by  DDR&E  with  the  advice  of  the  services. 
Owing  to  a  number  of  difficulties  arising  in  the  nature  of  the  data 
requirements  and  the  development  status  of  several  of  the  systems  in 
the  original  list,  it  was  not  possible  to  accumulate  complete  and  con¬ 
sistent  data  for  all  of  the  systems.  In  the  course  of  the  study, 
however,  data  became  available  for  several  systems  not  in  the  original 
sample.  Useful  data  covering  a  total  of  24  major  systems  or  principal 

^Throughout  this  report  the  term  "program  growth"  will  be  used  to 
represent  the  differences  between  the  actual  cost,  schedule,  and  pc- 
formance  outcomes  of  a  program  and  the  predictions  made  at  the  beginning 
of  weapon  system  development. 
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subsystems  eventually  constituted  the  data  base  for  the  1960s  (see 
Table  1) .  As  can  be  seen  from  the  listing,  the  six  Army  systems 
include  missiles,  helicopters,  and  a  tank;  the  eight  Navy  systems 
include  aircraft,  torpedoes,  sonars,  and  a  sonobuoy;  and  the  ten  Air 
Force  programs  include  various  kinds  of  aircraft  and  missile  systems 
and  major  subsystems.  Although  this  sample  is  limited,  it  is  repre¬ 
sentative  of  the  types  of  major  systems  acquired  by  the  Department  of 
Defense  in  the  decade  of  the  1960s.  The  total  dollar  value  of  these 

A 

programs  is  in  excess  of  $23  billion.  Of  this  amount,  about  80  percent 
is  accounted  for  by  the  Air  Force  systems;  almost  50  percent  of  the 
total  is  attributable  to  the  cost  of  two  very  large  Air  Force  programs 
—  the  C-5A  and  the  F-lll. 

During  analysis  of  the  data,  the  initial  emphasis  was  on  measuring 
the  amount  of  cost  growth  experienced  in  each  program.  Cost  growth 
was  measured  in  terms  of  the  differences  between  the  actual  costs  and 
estimates  made  at  the  beginning  of  system  development,  with  results 
expressed  in  terms  of  a  "cost  factor"  —  the  ratio  of  the  actual  cost 
to  the  estimated  cost. 

Since  cost  factors  were  used  as  the  basis  for  describing  and 
comparing  the  cost  experiences  of  acquisition  programs,  two  essential 
points  concerning  their  limitations  must  be  considered.  First,  such 
ratios  capture  a  number  of  influences  on  program  outcome  but  do  not 
provide  information  sufficient  to  distinguish  among  these  influences. 

And  second,  cost  factors  alone  are  inadequate  for  making  cost  com¬ 
parisons  since  different  programs  may  be  more  or  less  difficult  to 
estimate  accurately  or  carry  out  successfully. 

A  cost  factor  captures  several  influences  on  real  or  apparent 
program  outcomes.  Optimism  in  cost  estimating  —  or,  as  it  is  fre¬ 
quently  categorized,  the  optimism  inherent  in  the  advocacy  process 
that  precedes  the  approval  of  new  system  developments  —  is  one 

l/t 

Since  cost  data  were  not  available  for  all  systems,  it  is  dif¬ 
ficult  to  estimate  the  total  dollar  coverage. 
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Table  1 

SYSTEM  COVERAGE 


ARMY 


NAVY 


AIR  FORCE 


Cost,  Schedule,  and  Performance 

Pershing  I  OV-lOA 

Pershing  IA  DIFAR 

OH-6  A  (Hughes) 


Cost  and  Schedule  Only 


Schedule  and  Performance  Only 
Sheridan 
Cheyenne 
Lance 


A-7Ea 

SQS-26AXa 

SQS-26CX 

MK-48  Mod  0 
A- 7  A 


Schedule  Only 


MK-48  Mod  1 


Performance  Only 


F-lll 

C-5A 

C-141 

Titan  III-C 

Minuteman  II 

Airborne  Command  Post 

Minuteman  II 

Guidance  and  Control 


A-7Da 


XC-142 


Sprint 


SRAM 


aOnly  actual  schedule  data  were  available  for  these  systems.  Without 
appropriate  predicted  milestone  dates,  no  schedule  factors  could  be  gener¬ 
ated  for  inclusion  in  Figure  1. 
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obvious  contributor  to  the  difference  between  actual  and  estimated 
costs.  Imprecision  in  the  cost  estimating  process  itself  is  also 
reflected  in  the  cost  factor.  In  addition  to  these  influences,  the 
cost  factor  also  captures  cost  growth  that  occurs  as  a  result  of  scope 
change  and  technological  uncertainty.  The  term  "scope  change"  is  used 
here  to  identify  changes  in  program  goals  or  specifications  after  the 
start  of  development.  Scope  change  can  include  any  fundamental  change 
in  program  objectives  except  changes  in  the  quantities  of  systems 
ordered.  Technological  uncertainty  —  which  may  be,  in  part,  another 
aspect  of  optimism  —  arises  in  the  expectation  that  technical  problems 
can  be  resolved  without  causing  a  given  program  to  consume  more  than 
the  total  of  resources  programmed  at  the  time  system  development  began. 
Finally,  a  cost  factor  also  captures  inef f iciences  in  the  development 
process,  whether  they  arise  from  erroneous  development  strategies,  in¬ 
appropriate  contract  arrangements,  or  any  of  a  multitude  of  conceivable 
management  inefficiencies  on  the  part  of  either  the  customer  or  the 
supplier.  Cost  factors  alone  provide  insufficient  information  to  dis¬ 
tinguish  among  these  contributors. 

Therefore,  cost  factors  alone  should  not  be  used  to  compare  cost 
experience  of  different  programs.  A  comparison  might  be  made,  for 


A 

If  cost  factors  are  to  be  used  to  measure  cost  growth,  they  should 
be  calculated  from  estimates  that  have  not  been  inflated  or  understated 
in  response  to  such  circumstances  as  funding  pressures.  If  the  esti¬ 
mate  for  one  system  is  adjusted  because  of  anticipated  ease  or  difficulty 
in  obtaining  funding  while  the  estimate  for  another  system  is  not, 
comparison  of  the  cost  factors  derived  from  these  estimates  would  be 
meaningless.  Among  the  systems  in  the  sample  used  in  the  analysis 
reported  here,  there  is  no  reason  to  believe  that  there  were  important 
differences  in  the  pressures  to  obtain  funding.  It  is  thus  reasonable 
to  assume  that  cost  factors  can  be  used  as  a  basis  for  making  cost 
comparisons  among  these  systems. 

**Whenever  quantity  changes  occurred  during  a  program,  an  adjust¬ 
ment  was  made  to  reconcile  the  estimated  cost  with  the  actual  quantity 
produced.  Adjusting  the  estimated  cost  to  the  quantity  covered  by 
actual  program  cost  is  obviously  preferable  to  calculating  a  cost  fac¬ 
tor  based  on  different  quantities.  The  basic  data  also  were  adjusted 
to  compensate  for  inflation  that  occurred  during  the  decade.  The  means 
of  making  these  adjustments  have  been  reported  in  the  Appendix  of 
Harman  and  Henrichsen,  A  Methodology  for  Cost  Factor  Comparison  ana 
Prediction _,  RM-6269-ARPA,  August  1970. 
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example,  between  two  programs  having  cost  factors  of  1.2  and  1.6,  which 
would  describe,  respectively,  cost  growths  of  20  and  60  percent.  The 
cost  factor  of  1.2  could  represent  the  outcome  of  a  rather  simple  pro¬ 
gram  utilizing  off-the-shelf  components,  while  the  second  program  might 
involve  the  development  of  a  very  advanced  weapon  system  built  around 
several  major  subsystems  about  which  there  was  great  technical  uncer¬ 
tainty  at  the  time  program  development  began.  In  such  a  case,  the  cost 
factor  of  1.2  might  suggest  a  badly  structured,  inefficiently  managed 
program,  while  the  apparent  60  percent  cost  increase  of  the  other  pro¬ 
gram  might  in  retrospect  be  considered  money  well  spent.  Without  more 
information  on  the  programs  in  the  sample  than  their  respective  cost 
factors,  it  would  not  be  reasonable  to  say  very  much  about  individual 
program  effectiveness,  nor  would  it  be  possible  to  make  adequate  cost 
comparisons  among  systems.  Any  attempt  to  describe  the  contributors 
to  cost  growth  or  to  make  adequate  cost  comparisons  must  take  into 
account  the  influence  on  cost  of  other  program  attributes,  including 
especially  program  duration  and  technical  difficulty. 

The  cost  factors  for  programs  of  the  1960s  are  displayed  in  Table 
2.  The  original  expectation  that  the  estimates  contained  in  the  formal 
Technical  Development  Plan  might  be  used  as  a  baseline  for  1960s  program 
outcome  analysis  was  frustrated  when,  early  in  the  research  phase,  it 
was  discovered  that  Technical  Development  Plans  were  not  available  for 
a  number  of  systems  in  the  sample.  A  variety  of  other  sources  were 
used  instead.*  Nonetheless,  every  effort  was  made  to  obtain,  as  the 
earliest  benchmark,  an  estimate  valid  at  the  time  the  DoD  approved  the 
start  of  weapon  system  development.  The  underlying  assumption  was 
that  at  such  a  point  in  time,  the  cost,  schedule,  and  system  performance 
requirements  of  the  program  had  been  extensively  reviewed  by  the  re¬ 
sponsible  developers,  by  the  using  service,  and  by  approval  authorities 
in  the  Department  of  Defense. 

In  general,  program  cost  growth  for  the  1960s  (corrected  for 
inflation)  has  averaged  about  40  percent.  The  cost  growth  in  the 

*The  sources  of  the  data  actually  used  in  the  research  have  been 
reported  in  Harman  and  Henrichsen,  RM-6269-ARPA,  Appendix. 
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Table  2 

DEFLATED  COST  FACTORS  —  THE  19  60s  SYSTEMS 
(total  program  costs  as  of  mid-FYl970) 


Air  Force 


Navy 


Army 


Sys  tem 


Factor  System 


Factor 


Sys  tem 


Factor 


F-lll 

2.07 

A-7E 

1.40 

0H-6A  (Hughes) 

1.09 

1.95 

2.02 

OV-10A 

1.10 

Pershing  I 

1.12 

1.41 

SQS-26AX 

2.34 

1.01 

C-5A 

1.36 

Pershing  IA 

1.07 

1.38 

SQS-26CX 

1.55 

1.03 

1.09 

DIFAR 

2.05 

C-141 

1.16 

1.04 

1.41 

A-7D 

1.23 

Minuteman  II 
Airborne 

Command  Post  1.12 
1.28 

Minuteman  II 
Guidance  & 

Control  1.60 

Titan  III-C  1.06 


The  listing  of  multiple  cost  factors  for  individual  programs  indi¬ 
cates  that  different  "initial  estimates"  were  made  at  various  times 
between  program  conception  and  program  completion.  The  method  used 
to  reconcile  such  data  is  described  in  Harman  and  Henrichsen,  RM-6269- 
ARPA. 
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development  phases  of  programs  is  very  difficult  to  establish  but  seems 

A 

to  average  50  to  100  percent.  The  difficulty  of  establishing  precise 
development  cost  factors  arises  very  largely  from  the  cost  reporting 
system  used  by  the  services  during  most  of  the  1960s.  If  the  develop¬ 
ment  process  is  assumed  to  be  that  effort  required  to  accumulate  the 
information  necessary  to  begin  series  production  of  an  operationally 
acceptable  system,  then  some  portion  of  the  money  actually  spent  in 
the  period  after  development  formally  ended  (that  is,  work  required 
to  make  the  initially  produced  system  satisfy  real  operational  require¬ 
ments)  should  be  charged  as  part  of  development  cost.  In  most  cases, 
it  proved  either  difficult  or  impossible  to  obtain  an  accurate  account¬ 
ing  of  the  sums  spent  on  development. 

Schedule  and  performance  factors  were  also  considered  in  the 
analysis.  Characteristically,  there  was  an  average  15  percent  slip¬ 
page  in  schedules  in  the  course  of  a  program.  Although  the  mean  of 
the  performance  factors  was  about  1.0,  the  factors  extended  over  a 
range  of  about  0.5  to  2.0.  The  frequency  distributions  of  these  three 
elements  for  the  1960s  systems  are  indicated  in  Fig.  1.  The  cost, 
schedule,  and  performance  factors  are  displayed  so  that  all  points 
plotted  to  the  left  of  unity  represent  outcomes  better  than  predicted, 
and  all  points  plotted  to  the  right  of  unity  represent  outcomes  worse 
than  predicted. 

A  comparison  of  the  cost,  schedule,  and  performance  factors  sug¬ 
gests  that  performance  was  treated  as  the  dominant  system  development 
objective  and  that  schedule  goals  held  a  very  slightly  lower  priority. 


Questionnaire  returns  for  20  of  the  systems  in  the  original 
sample  indicated  a  mean  development  cost  factor  of  1.92  and  a  range 
from  0.98  to  5.57.  However,  there  is  great  imprecision  in  the  re¬ 
sponses,  arising  largely  from  data  inadequacies  and  definitional 
problems . 

**Some  performance  ratios  were  inverted  in  order  to  make  them 
consistent  with  the  above  description.  This  was  necessary  since,  in 
the  case  of  performance  factors,  a  desirable  outcome  sometimes  results 
in  an  actual/estimate  ratio  being  greater  than  unity  (for  example, 
missile  range) . 


Frequency 


Actual -to-estimated  ratio 


The  frequency  distributions  have  been  normalized  so  that  if  the  heights 
of  the  bars  for  any  factor  are  totaled,  the  sum  will  be  unity.  The  vertical 
axis  gives  the  proportion  of  factors  that  fall  within  any  given  interval .  The 
orginal  data  provide  more  schedule  and  performance  factors  than  cost  factors. 
This  anomaly  is  explained  by  the  fact  that  a  variety  of  milestones,  m  addi¬ 
tion  to  initial  operational  delivery,  were  used  in  the  schedule  analysis.  For 
the  performance  analysis,  the  project  officers  who  provided  the  original  da  a 
were  asked  to  identify  four  or  five  principal  performance  parameters  crucial 
to  their  systems.  This  was  necessary  to  permit  inclusion  of  performance 
parameters  for  a  wide  variety  of  systems. 

pig  j_— Frequency  distribution  of  cost,  schedule, 
and  performance  factors 
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Cost  increases  seem  to  have  been  accepted  in  order  to  meet  performance 
and  schedule  goals.  There  are  two  sources  of  support  for  this  inter¬ 
pretation.  First,  on  the  average,  performance  objectives  were  met 
while  schedules  were  allowed  to  slip  somewhat  and  costs  even  more. 

Also,  it  is  clear  from  the  entire  distribution  of  outcomes  that  per¬ 
formance  was  frequently  attained  or  exceeded;  to  a  lesser  extent  the 
same  is  true  of  schedules;  but  costs  have  invariably  been  higher  than 
predicted . 

The  program  outcomes  of  the  1960s  reported  in  Table  2  were  com¬ 
pared  with  a  data  base  of  21  Air  Force  aircraft  and  missile  programs 
of  the  1950s.  These  data  were  originally  compiled  by  Eugene  Brussell 
of  Rand  between  1957  and  1961.  They  were  initially  used  in  weapons 
acquisition  studies  by  A.  W.  Marshall  and  W.  H.  Meckling  in  1959 
and  subsequently  by  B.  H.  Klein  and  R.  Summers  in  a  series  of  studies 
completed  in  the  early  1960s.  Parallel  but  entirely  separate  studies 

of  the  characteristics  of  the  weapon  system  acquisition  process  of  the 
1950s  were  conducted  by  M.  J.  Peck  and  F.  M.  Scherer  of  Harvard  Univer- 
sity  in  the  late  1950s  and  early  1960s.  Basically,  it  is  the 

findings  of  these  studies  that  have  most  often  been  cited  by  critics 
of  the  present  acquisition  process  in  attributing  highly  inefficient 
development  practices  to  the  1950s  and  in  assuming  that  the  practices 


See  Perry  et  al.3  System  Acquisition  Experzence3  RM-6072-PR, 
November  1969,  for  a  more  detailed  discussion  of  performance  factors. 

A.  W.  Marshall  and  W.  H.  Meckling,  Predictability  of  the  Costs3 
Iime3  and  Success  of  Development ,  The  Rand  Corporation,  P-1821, 

December  1959,  also  in  The  Rate  and  Direction  of  Inventive  Activity 3 

Princeton  University  Press,  Princeton,  1962. 

***Burton  H.  Klein,  "The  Decision  Making  Problem  in  Development," 
in  The  Rate  and  Direction  of  Inventive  Activity;  Robert  Summers,  Cost 
Estimates  as  Predictors  of  Actual  Weapons  Costs:  A  Study  of  Mac  or 
Hardware  Articles 3  RM-3061-PR,  The  Rand  Corporation,  March  1965;  also 
in  T.  Marschak,  T.  K.  Glennan,  Jr.,  and  Robert  Summers,  Strategy  for 
R&D:  Studies  in  the  Microeconomics  of  Development 3  Springer-Verlag, 

New  York,  1967. 

****M.  J.  Peck  and  F.  M.  Scherer,  The  Weapons  Acquisition  Process: 
An  Economic  Analysis 3  Harvard  University  Press,  Cambridge,  Massachusetts, 
1962;  and  F.  M.  Scherer,  The  Weapons  Acquisition  Process:  Economic  In¬ 
centives  3  Harvard  University  Press,  Cambridge,  Massachusetts,  1964. 
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followed  in  certain  prominent  programs  of  the  1960s  were  at  the  same 
general  level  of  ineffectiveness.  In  all  cases  the  earlier  researchers 
were  concerned  with  cost  factors. 

Ail  of  the  researchers  acknowledged  two  difficulties  that  in  some 
respects  limited  the  usefulness  of  their  findings.  First,  in  the  1950s 
(and  particularly  in  the  early  1950s)  analytical  methods  of  estimating 
probable  program  costs  were  in  a  very  primitive  stage  of  development, 
and  cost  estimates  therefore  tended  to  be  imprecise.  A  further  dif¬ 
ficulty  was  that  many  of  the  systems  to  which  the  cost  estimation 
techniques  were  applied  in  the  1950s  were  more  complex  and  more  tech¬ 
nically  difficult  than  the  systems  used  as  a  data  base  for  early  cost 
estimation  models  —  the  systems  with  which  the  Air  Force  had  experience 
shortly  after  World  War  II.  Comparisons  were  awkward  and  therefore  not 
always  acceptable  to  critical  audiences.  A  second  major  complication 
was  the  difficulty  of  obtaining  initial  estimates  that  were  consistent 
with  one  another.  There  was  no  single  point  in  the  programs  of  the 
1950s  at  which  a  cost  estimate  was  systematically  reviewed  and  in  some 
sense  approved  by  officials  who  were  responsible  for  authorizing  the 
start  of  system  development.  Researchers  had  therefore  to  attempt  to 
reconcile  several  sets  of  estimates  made  at  various  times  throughout 
the  planning  and  development  phase  of  each  program.  Cost  factors  for 
the  1950s  programs  are  presented  in  Table  3. 

The  methodology  used  in  comparing  cost  factors  of  one  decade  with 
those  of  another  decade  is  fully  documented  in  a  recent  Memorandum 
so  only  a  few  salient  features  need  be  summarized  here.  The  most  im¬ 
portant  element  of  the  analysis  was  the  introduction  of  a  measure  of 
technical  difficulty.  Early  in  the  course  of  conducting  this  research 
it  became  apparent  that  some  measure  of  the  technical  difficulty  of 
individual  programs  had  to  be  included  in  order  to  make  meaningful  com¬ 
parisons  among  programs  of  markedly  different  technical  scope.  For 
the  purposes  of  the  initial  analysis,  the  programs  in  the  1950s  sample 
and  the  programs  in  the  1960s  sample  were  rated  on  a  scale  ranging  from 
1  to  20,  the  numerical  value  in  each  instance  being  intended  to 

*Harman  and  Henrichsen,  RM-6269-ARPA. 
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Table  3 

DEFLATED  COST  FACTORS  —  THE  1950s  SYSTEMS 


System  Factor  System 


F-102/106 

4.06 

2.46 

1.49 

1.30 

2.35 

B-58 

1.30 

B-52 

F-101 

.57 

1.12 

F-100 

1.20 

F-94C 

2.56 

B-47 

F-89 

2.04 

1.52 

C-133 

F-86D 

.78 

C-130A 

F-86A 

.93 

KC-135 

F-84F 

2.02 

F-84C 

1.55 

Factor  System  Factor 


4.00  IRBM  Thor  1.33 

5.  it) 

3.64  Snark  3.10 

1.95  1*17 

1.53 

2.62 

1.33  I CBM  Titan  1.00 

1.44 

.97  I CBM  Atlas  .77 

1.46  -82 

.87  1-32 

.85 

1.25  *88 

.70 

1.55  Falcon  2.52 

3.80 

1.55  2.09 

.51 

1.49 

Bomarc  7 .10 

.81  3.90 

.92  3.48 

.80  2-22 

1.49 

1.16 

1.00 


The  listing  of  multiple  cost  factors  for  individual  programs 
indicates  that  different  "initial  estimates"  were  made  at  various 
times  between  program  conception  and  program  completion.  The  method 
used  to  reconcile  such  data  is  described  in  Harman  and  Henrichsen, 
RM-6269-ARPA. 
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represent  the  magnitude  of  technical  advance  that  was  sought  at  the 
time  of  program  approval.  In  the  initial  research  such  values  were 
assigned  subjectively,  based  on  assessments  obtained  from  members  of 
the  Rand  staff  who  were  cognizant  of  the  characteristics  and  attributes 
of  individual  weapon  systems.  (Ratings  for  the  1950s  systems  were 
based  on  a  survey  conducted  by  Eugene  Brussell  and  employed  by  Marshall 
and  Meckling,  Klein,  and  Summers.)  This  numerical  value,  which  became 
known  as  the  A-f actor,  was  introduced  in  the  cost  factor  model  developed 
by  Harman. 

In  an  effort  to  expand  the  A-factor  data  base,  a  larger  number 
of  experts,  both  within  and  outside  of  Rand,  in  this  country  and  abroad, 
subsequently  contributed  subjective  ratings  for  a  rather  large  number 

of  systems  characteristic  of  the  1950s  and  1960s.  The  subjective 

* 

ratings  in  the  expanded  sample  are  reproduced  in  Fig.  2.  As  a  means 
of  distinguishing  between  the  various  categories  of  technical  advance 
sought  at  the  onset  of  development  in  a  given  program,  generalized 
definitions  were  adopted  for  nine  categories  ranging  from  2  to  18  on 
the  numerical  scale. 

The  other  principal  determinant  of  cost  factor  differences  between 
programs,  as  indicated  at  this  stage  of  the  research,  was  program  length. 
For  the  purposes  of  this  analysis,  an  effort  was  made  to  measure  the 
duration  of  a  program  between  the  point  at  which  the  service  was  author¬ 
ized  to  proceed  with  full-scale  weapon  system  development  and  the  point 
at  which  the  development  was  essentially  completed.  However,  differ¬ 
ences  in  definition  among  the  services  and  the  rather  wide  variety  of 
systems  with  quite  different  operating  and  production  characteristics 
included  in  the  sample  made  it  advisable  to  avoid  the  end-of-program 
indicator  most  frequently  used  in  past  studies  of  this  sort:  initial 
operational  capability  (which  ordinarily  means  the  point  at  which  an 
operationally  useful  quantity  of  systems  has  been  delivered  to  the 
using  service).  Instead,  the  dating  of  the  development  program  "end" 

*The  data  shown  were  accumulated  during  several  surveys  conducted 
over  many  months,  and  the  procedures  varied  in  detail  from  time  to 
time,  so  that  all  data  points  may  not  be  strictly  comparable. 


BASICALLY  NEW  AND  RADICALLY 
DIFFERENT  SYSTEM  DESIGN 


NEW  TECHNOLOGY  REQUIRED, 

AND  NEW  DESIGN  (SUBSYSTEMS) 
TO  MEET  PERFORMANCE 
SPECIFICATIONS 


NEW  TECHNOLOGY  MUST  BE 
DEVELOPED  TO  MEET  SYSTEM  NEEDS 


SEVERAL  MAJOR  SYSTEM  ELEMENTS 
REQUIRE  MAJOR  IMPROVEMENT 
(10-20%) 


AT  LEAST  ONE  MAJOR  SYSTEM 
ELEMENT  REQUIRES  MAJOR 
IMPROVEMENT 


SEVERAL  MAJOR  SYSTEM  ELEMENTS 
REQUIRE  IMPROVEMENT  AND 
INTEGRATION 


AT  LEAST  ONE  MAJOR  SYSTEM 
ELEMENT  REQUIRES  IMPROVEMENT 
(ENGINE,  AVIONICS,  ETC.) 
(5-10%  PERFORMANCE) 


CONTEMPORARY  TECHNOLOGY 
REQUIRING  INTEGRATION 


NO  NEW  KNOWLEDGE  REQUIRED 
(SHELF  ITEMS) 
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20  r- 


18 


16 


14 


12 


10 


•  NAVAHO 

•  US/SST 

•  SR -71 

•  B  -  47  *B-58 

•  F-lll 

•  ATLAS  •F-86D 

•  FALCON 

•  THOR 

•  KESTREL  VTOL 

FIGHTER 

•  HUNTER  (U  K ) 

•  VIGGEN 

•  F- 105 

•  F- 101 

•  F-84C  •  F-94C 

•  HARRIER  VTOL 

FIGHTER 

•  AG ENA  D 

•  LOH  (HUGHES) 

•  C-141 


•  CONCORDE 

•  X-l  RESEARCH 

AIRCRAFT 

•  BOMARC  •  SNARK 

•  F  - 102 

•  F-86A 

•  LIGHTNING  P-1  (UK) 

•  F-100 

•  F-89  •  B-52 

•  TITAN  I 

•  MIRAGE  IV 

•  JAVELIN  (UK) 

•  F-84F 

•  SCIMITAR  (‘47 ) 

•  TITAN  III -C 

•  DRAKEN 

•  C  -5  A  •  KC  - 1 35 

•  MM  ll-ACP 

•  C  - 133 

•  A-7D  •  A-7E 

•  MIRAGE  III 

•  MIRAGE  V 


•  JETSTAR  (PROTOTYPE) 


•  F-104J 


•  P-1127  (VTOL  FIGHTER) 

•  DISCOVERER 

•  GEMINI 

•  F-  106 

•  VULCAN  (U  K ) 

•  POLARIS  I  •VICTOR  (UK) 

•  MM  II  -  G  4C 

•  SWIFT  I  (UK) 

•  CANBERRA  I 

•  OV-10  •  PERSHING  I 

•  BUCCANEER  (UK) 

•  MIRAGE  G-4  •  VALIANT  (UK) 

•  GNAT  I  (U  K ) 

•  LIGHTNING  MK  III  (UK) 

•  C- 130  •  PERSHING  I A 


•  SABRELINER 


Fig.  2 — Technological  advance  ratings 
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was  taken  to  be  at  the  time  of  acceptance  of  the  first  operational 
item  delivered  to  the  using  service. 

It  does  no  great  violence  to  conventional  wisdom  to  suggest  that 
programs  with  a  high  content  of  technical  risk  and  of  considerable 
duration  would  be  more  difficult  to  predict  and  control  than  straight¬ 
forward  short-term  programs.  The  analysis  of  programs  spanning  two 
decades  transforms  such  conventional  wisdom  into  supportable  findings. 
Longer  programs  and  those  requiring  larger  technological  advances 
usually  have  higher  cost  factors,  indicating  poorer  predictability 
of  program  outcomes  for  such  systems. 

A  product  of  the  analysis  was  the  conclusion  that  the  prediction 
and  control  of  system  acquisition  programs  had  not  appreciably  improved 
in  the  1960s  over  the  1950s,  although  program  outcomes  tended  to  deviate 
less  from  program  predictions  in  the  1960s  because  lower  technological 
advances  and  shorter  programs  were  characteristic  of  that  decade. 

That  point  may  be  illustrated  by  observing  that  the  typical  A-factor 
value  of  programs  of  the  1950s  was  12.2  while  the  average  A-factor 
value  for  the  1960-decade  systems  in  the  sample  was  8.9.  Such  a  com¬ 
parison  suggests  very  strongly  that  the  principal  effect  of  the  program 
review  processes  introduced  in  1961  and  honored  thereafter  was  to 
screen  out  some  of  the  higher  risk  programs  that  might  have  been 
approved  for  development  in  an  earlier  era,  with  resulting  extreme 
departures  from  predicted  outcomes. 

It  is  also  apparent  that  when  a  full  account  is  taken  of  the  pro¬ 
gram  characteristics  mentioned  above,  the  popular  impression  that  pro¬ 
grams  of  the  1950s  tended  to  experience  cost  growth  of  200  percent  or 
more  in  the  ordinary  course  of  events  may  not  be  an  accurate  representa¬ 
tion  of  what  actually  occurred.  The  distribution  of  outcomes  for  the 
1950s  is,  as  revealed  in  the  tables  above,  somewhat  broader  than  for 
the  1960s.  But  for  programs  of  comparable  length  and  technical  dif¬ 
ficulty,  differences  in  program  outcomes  for  the  two  decades  are  not 
statistically  significant.  Further,  given  the  assumption  that  programs 

For  additional  details  on  the  comparison  between  the  decades, 
see  Harman  and  Henrichsen,  RM-6269-ARPA. 
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of  the  1970s  will  be  conducted  very  much  along  the  lines  of  those  of 
the  late  1960s,  future  cost  uncertainty  will  still  be  rather  large, 
especially  for  programs  that  will  require  major  technological  advances 
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II .  THE  SOURCES  OF  PROGRAM  GROWTH  AND  THE  PREDICTABILITY 

- : -  OF  PROGRAM  OUTCOMES 

RECENT  EXPERIENCE 

Examination  of  the  development  histories  of  several  major  Air 
Force  programs  provides  a  general  explanation  for  the  causes  of  cost 
growth.  The  three  broad  candidates  for  cost  growth  responsibility 
were  identified  earlier:  technical  uncertainty,  scope  change,  and 
cost  estimating  error.  Obviously,  inefficiencies  in  program  manage¬ 
ment  or  the  selection  of  an  inappropriate  program  strategy  would  tend 
to  be  captured  by  one  of  these  three  broad  categories.  About  one-third 
of  observed  cost  growth  and  much  of  the  deviation  of  system  performance 
from  that  initially  anticipated  appear  to  be  attributable  to  technical 
uncertainty.  So  also  is  some  small  part  of  the  observed  schedule  slip¬ 
page.  However,  approximately  one-half  of  observed  cost  growth  and  an 
additional  part  of  schedule  slip  appear  to  be  directly  attributable  to 
scope  change  (that  is,  changes  in  program  objectives  imposed  on  the 
program  after  the  start  of  weapon  system  development) .  The  residual 
cost  growth  can  be  attributed  to  estimating  inaccuracies. 

In  an  era  of  declining  budgets,  the  consequence  of  spending  a 
great  deal  more  money  than  had  originally  been  planned  for  a  given 
program  may  be  a  reduction  in  the  quantity  of  items  ultimately  pur¬ 
chased  for  the  inventory.  Characteristically,  in  the  1960s  the  total 
funding  requirements  projected  at  the  onset  of  a  program  have  been 
exceeded  only  modestly;  when  development  costs  and  unit  costs  proved 
to  be  higher  than  anticipated,  the  need  for  increased  total  funding 
was  often  avoided  by  a  cutback  in  production  quantities.  The  negative 
consequences  of  this  practice  for  planned  force  structure  are  obvious. 

Of  the  sources  of  cost  growth  cited  above,  estimation  inaccuracies 
have  the  least  impact.  Only  about  15  percent  of  the  observed  cost 
growth  (the  residual  mentioned  above)  can  be  assigned  to  the  inherent 
Imprecision  of  present  cost  estimating  procedures.  That  is  not  a 
small  error.  Nevertheless,  if  this  general  distribution  of  the  sources 
of  program  deviations  is  valid  (and  no  evidence  that  strongly  contradicts 
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these  findings  has  yet  been  unearthed) ,  reducing  the  inaccuracy  of 
present  cost  estimating  procedures  by  half,  which  by  general  consent 
would  represent  a  very  substantial  improvement,  probably  would  have 
no  clearly  detectable  effect  on  the  predictability  of  program  outcomes 


The  effects  of  scope  change  are  sometimes  difficult  to  distinguish 
from  the  effects  of  changes  resulting  from  unanticipated  technical 
difficulties.  However,  an  examination  of  the  details  of  several  major 
programs  indicates  that  scope  change  generally  entails  levying  more 
demanding  performance  requirements  on  individual  systems  and  that 
such  change  ordinarily  imposes  additional  costs  on  an  acquisition  pro¬ 
gram.  It  is  necessary  to  recognize,  however,  that  it  is  in  the  nature 
of  major  programs  to  change  objectives  as  time  passes:  requirements 
are  refined  to  reflect  new  or  updated  threat  assessments,  alterations 
in  national  strategy  cause  mission  requirements  for  individual  systems 
to  expand,  and  new  and  promising  technology  influences  systems  in 
development.  Thus,  for  example,  the  complexity  of  avionics  installa¬ 
tions  and  the  requirement  for  greater  avionics  responsiveness  to  a 
larger  variety  of  operational  situations  tend  to  induce  a  regular 
growth  in  demands  for  avionics  performance  as  a  program  proceeds . 

Accuracy  and  reliability  requirements  for  missiles,  aerodynamics  per¬ 
formance  parameters  for  aircraft  and  helicopters,  and  similar  perform¬ 
ance  functions  of  other  weapon  systems  are  frequently  mane  more  demanding 
Such  alterations  in  program  objectives  account  for  the  majority  of 
scope  change  effects. 


The  influence  of  the  third  source  of  cost  growth,  technical  uncer 
tainty,  is  illustrated  by  Fig.  3,  which  shows  the  dependence  of  cost 
factors  on  program  duration  and  technical  difficulty.  This  figure 
displays  the  cost  factor  plotted  against  program  length,  for  three 
ranges  of  technological  advance.**  Programs  having  A-f actor  values 


‘Notwithstanding  this  indication,  work  on  f 

of  cost  estimating  processes  still  is  xn  progress,  both  at  Rand  and  el 
Shere  If  oSer  sources  of  program  growth  can  be  identified  and  ccntroU^ 
Ihen  rte  accuracy  of  the  cost  estimating  process  could  well  become  a 
more  significant  consideration. 

**See  Harman  and  Henrichsen,  RM-6269-ARPA  for  details  of  the  regres- 


sion  analysis. 
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of  less  than  about  12  tend  to  be  reasonably  predictable,  although  the 
distribution  of  outcomes  begins  to  widen  for  programs  with  A-factor 
values  larger  than  8.  For  programs  with  A-factors  above  12,  the  pre¬ 
dictability  of  program  outcomes  lessens  appreciably,  and  the  worsening 
effect  is  not  on  a  linear  scale.  As  the  A-factor  value  increases  and 
program  duration  extends  past  60  months,  the  cost  outcome  of  a  given 
program  becomes  so  unpredictable  that  initial  cost  estimates  are  very 

nearly  worthless. 

The  model  could  also  be  used  to  indicate  the  probable  extent  of 
cost  growth  for  future  systems.  The  implication  of  the  initial  research 
was  that  if  the  A-factor  of  a  new  system  could  be  accurately  assessed 
and  appropriate  account  taken  of  it  at  the  time  estimates  were  made, 
the  extent  of  program  uncertainty  might  reasonably  be  postulated. 

Given  the  assumption  that  future  programs  being  assessed  will  be 
planned  and  conducted  very  much  in  the  fashion  of  the  programs  in 
recent  experience,  from  which  the  data  are  drawn,  it  is  theoretically 
feasible  to  predict  the  probable  cost  growth  of  a  program  in  terms  of 
its  duration  and  the  degree  of  technical  advance  it  encompasses.  First, 
it  is  necessary  to  assess  the  level  of  technical  difficulty  encountered 
by  weapon  system  development  programs  in  the  past.  Such  an  assessment 
can  be  applied  to  entire  systems  or  to  subsystems.  Then,  as  a  second 
step,  it  is  possible  to  measure  the  total  program  cost  growth  and  the 
typical  schedule  change  incurred  in  relevant  past  programs  and  to  relate 
that  measurement  to  the  assessed  technical  difficulty  of  a  specific 
program  being  analyzed.  Using  the  cost  factor  model  described  in  Sec¬ 
tion  I  as  a  basis,  the  probable  cost  growth  of  a  proposed  system  (mea¬ 
sured  by  its  cost  factor)  can  then  be  predicted. 

Such  a  process  can  aid  in  anticipating  prospective  cost  growth 
only  to  the  extent  that  the  development  strategy  and  management  style 
of  the  program  being  examined  are  fundamentally  similar  to  those  repre¬ 
sented  in  the  data  base.  Were  a  substantially  different  strategy  used 
for  future  programs,  as  might  occur  if  new  approaches  to  weapon  system 
acquisition  were  required  by  DoD  or  forced  by  budgetary  pressures,  the 
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procedures  described  here  for  estimating  program  outcomes  could  become 
unreliable.  Introducing  parameters  representative  of  each  of  several 
basic  strategies  might  tend  to  compensate  for  such  uncertainty,  but  as 
yet  there  is  no  way  to  define  such  strategy  parameters  with  precision. 

A  second  consideration  when  using  such  a  procedure  in  a  predictive 
mode  is  the  irresponsibility  of  assuming  that  the  cost  estimate  for  a 
given  system  should  be  adjusted  upward  in  some  arbitrary  way  solely  be¬ 
cause  the  system  being  examined  has  a  "high"  A-factor  value.  Apparent 
cost  growth  would  be  limited  by  such  a  strategy,  but  the  goal  of  most 
program  managers  is  to  control  cost  rather  than  to  minimize  apparent 
cost  growth.  Further,  the  implication  of  a  "high"  A-factor  may  be  that 
the  system  being  analyzed  will  cost  more  —  and  perhaps  a  great  deal 
more  —  than  generally  comparable  programs.  Yet,  neither  the  original 
cost  estimate  nor  a  cost  estimate  adjusted  by  an  A-factor  corrective  is 
necessarily  a  "should  cost"  estimate.  Rather,  each  indicates  probable 
cost,  given  certain  basic  assumptions  about  program  length,  the  techno¬ 
logical  content  of  the  program,  and  acquisition  style.  Should  any  esti¬ 
mate,  however  obtained,  strongly  suggest  that  program  costs  will  be 
unacceptably  high,  an  appropriate  response  would  be  to  change  one  or 
more  of  the  variables  —  program  style,  or  level  of  technical  advance 
sought,  for  example.  Alternatively,  if  other  variables  must  remain 
constant,  for  any  of  several  possible  reasons  (such  as  urgency  of  the 
threat) ,  it  would  be  no  more  than  ordinary  common  sense  to  acknowledge 
that  program  costs  may  be  very  high  —  and  to  adjust  budgets  accord¬ 
ingly.  It  is  reasonable  also  to  infer  that  a  reduction  in  the  value 
of  a  technical  advance  factor  could  be  obtained  by  demonstrating  in 
some  relatively  inexpensive  way  that  the  desired  technical  advance 
could  actually  be  achieved.  Finally,  there  is  no  evidence  that  imposing 
additional  management  controls  of  the  sort  contrived  to  counteract 
some  of  the  unfavorable  program  outcomes  of  the  1960s  will  appreciably 
reduce  the  probability  of  an  undesired  program  outcome  in  the  1970s. 

A  parametric  cost  estimating  model  that  includes  an  A-factor  as 
one  of  the  variables  has  several  attractions.  But  before  an  attempt 
is  made  to  develop  such  a  model,  it  would  be  advisable  to  generate 
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A-factor  values  that  are  not  highly  dependent  on  the  candor  and  objec¬ 
tivity  of  the  estimators  —  by  some  technique  not  subject  to  the 
appraisers'  knowledge  of  past  program  outcomes.  The  underlying  assump- ^ 
tion  of  A-factor  analysis  is  that  the  assigned  value  reflects  an  objective 
appraisal  of  the  technical  advance  actually  sought  at  the  time  the 
program  began.  Thus  some  improved  measure  of  the  range  of  technical 
advance  sought  is  greatly  to  be  desired. 


SOME  PROCEDURAL  APPROACHES  TO  CONTROL  OF  COST  GROffljH 

Among  ways  of  decreasing  the  probability  of  large  cost  growth  in 
new  programs  are  some  potential  procedural  changes  of  rather  limited 
scope.  The  first  of  these  is  to  modernize  and  somewhat  improve  the 
accuracy  of  present  cost  estimating  models.  As  mentioned  above,  how¬ 
ever,  even  substantial  improvements  in  such  models  will  not  signifi¬ 
cantly  improve  the  predictability  of  costs.  A  second  approach  might 
be  to  increase  the  emphasis  on  the  cost  elements  in  contracting.  How¬ 
ever,  conventional  ways  of  utilizing  incentives  for  the  cost  elements 
in  system  contracting  do  not  appear  to  hold  much  promise. 

It  is  possible,  at  least  theoretically,  to  identify  early  indi¬ 
cators  of  impending  design  problems  during  development.  An  increase 
in  the  frequency  of  engineering  changes,  a  mid-program  shift  to  newer 
and  more  exotic  materials,  an  increase  in  the  number  of  engineers 
devoting  attention  to  some  technical  area,  or  any  of  several  other 
"symptoms”  might  be  used  in  this  fashion.  Evidence  that  such  evalua¬ 
tion  has  operational  utility  in  the  early  detection  of  design  and 
engineering  problems  is  lacking,  however.  One  difficulty  may  be  that 
current  data  reporting  systems  are  insensitive  to  such  indicators  — 


*The  possibility  that  useful  A-factor  values  can  be  anaiytical  y 
obtained  has  been  explored  on  a  modest  scale  since  ®  ^ 

this  line  of  research  has  promise,  no  conclusive  findings  have  yet 

emerged. 

**I.  N.  Fisher,  A  Reappraisal  of  Incentive  Contracting  Experience, 
RM-5700-PR  July  1968.  Some  additional  and  more  recent  work  a  ong 
C  lines  indicates  that  the  differences  in  Progr-  ~tc«e.  ha 
can  be  attributed  to  differences  in  contract  types  are  for  pract 
purposes  undetectable. 
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and  there  is  a  considerable  uncertainty  about  what  are  the  real  indi- 
* 

cators . 

One  of  the  difficulties  of  relying  on  development  program  trouble 
indicators  is  that  they  can  be  highly  unreliable  if  either  the  con¬ 
tractor  or  the  program  manager  wishes  to  disguise  his  problems  in  the 
expectation  of  being  able  to  resolve  them  before  they  attract  attention. 
Reliance  on  technical  milestone  indicators  of  the  sort  that  have  been 
used  to  "validate”  progress  in  recent  programs  may  be  unwise.  Indeed, 
the  practice  of  awarding  incentive  payments  because  a  program  has  passed 
a  technical  or  demonstration  milestone  may  in  some  cases  encourage  a 
contractor  to  demonstrate  some  feature  on  a  one-time  basis  even  though 
the  real  technical  capability  nominally  demonstrated  may  not  be  routinely 
or  regularly  achievable  at  all.  Measurements  of  sustained  performance 
(for  example,  the  50-hour  operating  requirement  imposed  on  jet  engines) 
may  represent  the  only  reliable  milestone  indicators  of  technical 
progress . 

Finally,  in  the  category  of  improved  program  control  procedures 
that  might  be  invoked  without  materially  altering  present  program 
structures  lies  the  possibility  of  devising  an  improved  method  for 
handling  scope  changes.  Evidence  obtained  from  case  histories  and 
from  examinations  of  scope  changes  that  have  affected  recent  programs 
indicates  that  the  impact  of  such  scope  change  on  program  outcomes  is 
not  ordinarily  assessed  on  a  total  program  basis.  Only  recently,  for 
example,  have  significant  cost  constraints  been  placed  on  the  engi¬ 
neering  change  process.  Although  there  would  appear  to  be  a  consider¬ 
able  area  for  improvements  in  the  control  of  scope  change,  minor  changes 
in  management  procedures  are  unlikely  to  create  any  major  improvement 
in  outcomes. 


The  hypothesis  that  probable  program  growth  can  be  foreseen 
toward  the  end  of  initial  hardware  test  phases  is  being  examined  as 
one  possible  way  of  identifying  impending  program  growth. 
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III.  SOME  ALTERNATIVE  ACQUISITION  STRATEGIES 

If  the  probable  outcome  of  a  1970s  program  conducted  in  the  pat¬ 
tern  of  typical  programs  of  the  1960s  is  likely  to  be  unacceptable  to 
the  sponsoring  service,  the  Department  of  Defense,  the  Congress,  and 
the  general  public,  is  it  reasonable  to  suggest  that  alternative  approaches 
to  the  acquisition  process  may  provide  some  relief?  An  answer  may  lie 
in  examination  of  recent  development  programs  that  did  not  conform  to 
’’typical"  patterns.  There  are  several  obvious  candidates  for  research 
in  this  area.  One  includes  Department  of  Defense  programs  exempted 
from  the  ordinary  controls  imposed  on  acquisition  processes  of  the 

1960s.  A  second  category  includes  a  range  of  development  programs 

* 

conducted  in  Europe  in  the  last  ten  years. 

In  the  category  of  programs  exempted  from  the  ordinary  controls 
imposed  by  the  Department  of  Defense  system  acquisition  process  generally 
favored  in  the  1960s,  it  is  difficult  to  find  examples  that  can  be 
cited  on  an  unclassified  basis.  However,  one  extremely  interesting 
case  is  the  Agena  D. 


AGENA  D 

Early  in  1961,  increased  activity  in  various  U.S.  space  programs 
and  dissatisfaction  with  the  performance  and  cost  of  available  space 
vehicles  induced  the  Air  Force  Space  Systems  Division  to  consider  means 
of  reducing  costs  and  increasing  launch  vehicle  flexibility  through 
standardization  of  what  was  then  called  the  Agena  B.  In  June  of  that 
year,  the  Space  Systems  Division  authorized  Lockheed  to  begin  the  design 


*Another  possible  candidate  is  civil  aviation  programs..  The 
obvious  area  for  inquiry  is  the  development  of  large  commercial . trans- 
port  aircraft.  Rand  has  completed  relatively  little  work  in  this  area; 
very  preliminary  research  suggests  that  there  is  cost  growth  on  the 
order  of  at  least  10  percent  and  schedule  slippage  of  10  percent  to 
15  percent.  There  are  also  performance  shortfalls  perhaps. as  large 
as  10  percent.  The  differences  between  actual  outcome  and  original, 
plan  are  clearly  smaller  for  commercial  aircraft  than  for  their  military 
counterparts,  but  no  striking  differences  in  performance  requirements 
or  technology  that  explain  the  differences  in  program  outcomes  can  be 
readily  identified. 
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of  a  standardized  Agena  space  vehicle  and  to  submit  a  proposal  for  its 
development  and  test.  Lockheed  responded  in  July,  and  on  25  August  ac¬ 
cepted  an  Air  Force  letter  contract  committing  the  firm  to  develop  and 
produce  what  was  to  become  known  as  the  Agena  D.  The  desired  vehicle 
was  to  be  used  in  a  number  of  different  space  programs  with  minimal 
modification;  to  lessen  development  time  and  limit  reliance  on  new  tech¬ 
nology,  Lockheed  was  to  use  available  Agena  B  technology  as  extensively 
as  possible.  The  contract  provided  for  the  delivery  of  twelve  flight 
vehicles  and  a  static  test  article,  the  creation  of  a  capability  for 
producing  five  vehicles  each  month,  and  the  establishment  of  a  price 
for  follow-on  procurement.  The  cost  of  the  development  effort  and  the 
required  vehicles  was  estimated  to  be  about  $60  million.  First  launch 
was  tentatively  scheduled  for  late  January  1963. 

Early  in  October  1961,  in  the  aftermath  of  a  briefing  on  stan¬ 
dardizing  the  Agena,  the  Undersecretary  of  the  Air  Force  concluded 
that  an  accelerated  schedule  and  perhaps  a  lower  development  cost  might 
result  from  adopting  an  approach  other  than  that  initially  proposed. 

He  appointed  a  special  committee  headed  by  C.  L.  Johnson,  Lockheed's 
Vice  President  for  Engineering,  to  "investigate  ways  and  means  of  pro¬ 
viding  a  more  reliable  Agena  on  an  accelerated  schedule."  Over  a  period 
of  ten  days  the  Johnson  committee  reviewed  the  Agena  proposal  and  Lock¬ 
heed's  capacity  for  accelerating  the  approved  schedule.  The  committee 
concluded  that  a  reliable  standard  Agena  could  actually  be  developed 
and  launched  by  June  1962  at  a  somewhat  lower  cost  than  initially  pro¬ 
posed,  provided  that  unusual  technical  and  contractual  relationships 
were  accepted  by  both  the  contractor  and  the  government.  The  Johnson 
committee  proposed  applying  techniques  derived  from  Johnson's  experience 
with  his  "skunk  works"  shop  at  Lockheed's  Burbank  plant.  To  insure 
compliance  by  both  the  contractor  and  the  government,  ground  rules  de¬ 
rived  from  that  experience  were  made  a  preamble  to  a  contractual  work 
statement  for  an  accelerated  Agena  D  program. 
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The  Undersecretary  approved  the  recommended  approach  and  directed 
that  appropriate  work  statements,  ground  rules,  and  contractual  provi¬ 
sions  be  drawn  up.  A  preliminary  agreement  was  signed  in  a  matter  of 
days.  By  20  November,  an  independent  program  office  reporting  directly 
to  the  Commander  of  the  Space  Systems  Division  had  been  established,  as 
had  a  counterpart  organization  at  Lockheed,  Sunnyvale.  When  the  details 
were  sorted  out,  the  Air  Force  project  officer  estimated  that  total 
funding  requirements  would  approximate  $32.9  million  dollars. 

The  resources  actually  expended  by  Lockheed  in  carrying  through 
the  agreement  are  summarized  in  Table  4.  The  "development  test  vehicle 
was  actually  completed  by  the  end  of  March  1962  and  was  immediately 
used  to  qualify  components  peculiar  to  the  various  applications  proposed 
for  the  Agena  D.  The  first  operationally  capable  Agena  was  delivered 
on  16  April  and  was  launched  on  27  June.  By  that  time,  all  four  in 
the  first  lot  of  Agena  Ds  had  been  delivered  (on  or  before  their  sched¬ 
uled  due-dates),  and  the  remaining  vehicles  were  on  schedule. 

A  standardized  production  configuration  was  adopted  and  approved 
by  the  Air  Force  in  September  1962,  and  production  of  a  larger  lot  of 
vehicles  began  immediately  after  completion  of  the  first  set  of  twelve. 
Of  the  first  24  Agena  Ds  launched  in  a  variety  of  programs,  21  were 
successfully  injected  into  orbit.  Total  program  costs  for  the  original 
twelve -vehicle  effort,  including  maximum  incentive  fees  on  performance, 
schedule,  and  cost  elements,  came  to  $31.7  million,  of  which  more  than 
$2  million  represented  fees  and  the  like.  The  engineering  work  needed 
to  satisfy  the  initial  development  objectives  proved  to  be  about  one- 
fourth  of  that  originally  postulated;  the  number  of  quality-control 
personnel  involved  was  reduced  from  an  anticipated  1200  to  a  total  of 
less  than  70;  and  tooling  costs  were  reduced  by  an  order  of  magnitude 
from  a  predicted  $2  million  to  $0.15  million.  Project  staffs,  both  in 
the  plant  and  the  SPO  at  the  Space  Systems  Division,  were  notably 
small.  Although  technical  reporting  was  minimal,  it  was  entirely 
adequate  when  judged  in  terms  of  work  accomplished.  Remarkably , 
only  about  10  percent  as  many  engineering  drawings  were  required  for 
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Table  4 

THE  AGENA  D  DEVELOPMENT  PROGRAM 


As  Planned 

As  Done 

Cost 

$60  million 

$32  million 

Time 

18  months 

9  months 

Drawings 

3900 

350 

Engineers  (factor) 

4 

1 

Quality  control  personnel 

1200 

69 

Product  improvement  program 

Yes 

No 

Tooling 

$2  million 

$0.15  million 

Drawing  release  lag 

30  days 

1  day 

Technical  progress  reporting 

"Normal" 

"Minimal" 

Project  office  staff  in  plant 

? 

4 

System  Project  Office  (SPO) 

1 

25 
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the  twelve-vehicle  Agena  D  program  as  would  ordinarily  ha've  been  de 
manded  for  a  program  conducted  by  the  rules  and  procedures  then  cus¬ 
tomary.  The  success  of  delivered  vehicles  in  meeting  program  objectives 
was  indisputable;  by  a  considerable  margin,  the  Agena  D  was  both  more 
mission-adaptable  and  more  reliable  than  its  Agena  B  predecessor  —  and 
it  cost  some  30  to  40  percent  less.  Reliability  was  at  least  twice  and 
perhaps  five  or  six  times  that  of  the  Agena  B. 

In  general,  the  working  interface  between  the  Air  Force  and 
Lockheed  and  the  effectiveness  of  the  relationship  were  appreciably 
better  in  the  view  of  most  observers  than  had  been  true  of  earlier 
work  of  the  same  general  nature.  Almost  precisely  the  work  projected 
in  the  original  proposal  had  been  completed,  the  cost  appeared  to  be 
about  one-half  of  that  originally  anticipated,  schedules  were  com¬ 
pressed,  and  the  performance  of  the  delivered  vehicle  was  appreciably 
superior  to  that  of  its  predecessor.  In  the  terms  ordinarily  used  to 

evaluate  a  program  of  this  sort,  the  Agena  D  development  effort  is 

•k 

difficult  to  fault. 

WEST  EUROPEAN  EXPERIENCE 

Two  of  the  authors  (Perry  and  Smith)  visited  several  major  European 
aircraft  firms  in  the  spring  of  1970  to  investigate  the  procedures  and 
achievements  characteristic  of  recent  aircraft  development  efforts. 

They  obtained  data  on  a  number  of  individual  aircraft  projects  from 
four  principal  European  manufacturers.  Two  were  British,  Hawker-Siddeley 
and  the  British  Aircraft  Corporation;  the  French  firm  was  Avions  Marcel 
Dassault;  SAAB,  the  only  Scandinavian  firm  now  developing  and  building 
military  aircraft,  was  the  Swedish.  Table  5  reproduces  a  selection 

*This  description  is  based  on  an  unpublished  Rand  case  study  of 
the  Agena  D  development  program  by  E.  0.  Johnson,  compiled  from  docu 
ments  in  historical  files  of  the  Air  Force  Space  and  Missile  Systems 
Organization.  Two  important  factors  of  the  foregoing  summary  should 
be  noted:  first,  the  later  Agena  D  production  program  did  not  follow 
the  pattern  of  the  original  development  effort;  second,  the  Agena  D 
rather  than  earlier  or  later  versions,  or  Agenas  used  by  NASA,  is  the 
topic  here. 
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of  data  obtained  during  the  trip.  For  the  most  part,  the  data  were 
furnished  by  the  aircraft  builders,  although  the  British  Ministry  of 
Technology  and  Ministry  of  Defense,  and  the  Aircraft  Materiel  Depart¬ 
ment  of  the  Swedish  Ministry  of  Defense  provided  corroborative  and 
supplemental  information  covering  aircraft  of  those  countries. 

A  simple  comparison  of  maximum  attainable  speeds  for  fighter  air¬ 
craft,  as  shown  in  the  first  line  of  Table  5,  is  of  course  not  a  complete 
indication  of  aircraft  performance;  but  it  serves  to  illustrate  that 
the  general  level  of  technology  exploited  by  major  European  aircraft 

builders  in  the  1960s  was  not  significantly  inferior  to  the  level  of 

* 

technology  achieved  by  major  American  aircraft  builders.  Of  the  six 
European  aircraft  chosen  for  illustrative  purposes  here,  three  have 
held  world  speed  marks  of  one  kind  or  another;  the  Harrier  is  the  only 
operational  vertical  takeoff  fighter  currently  in  use  anywhere  in  the 
world;  the  Viggen  is  a  unique  canard-configured  fighter  with  versatile 
performance;  the  Mirage-III  and  derivatives  of  it  have  been  the  basic 
fighter  aircraft  of  the  Israeli  Air  Force  and  have  been  used  widely  in 
Middle  East  combat  for  several  years;  and  the  Mirage- IV  is  a  70,000 
pound  aircraft  with  a  performance  not  greatly  different  from  that  of 
the  F-lll.  Certainly  there  are  performance  differences,  structural 
differences,  requirements  differences,  and  configuration  differences 
among  these  aircraft  and  between  such  aircraft  and  those  characteris¬ 
tically  used  in  the  United  States  for  the  past  eight  or  ten  years. 

But,  in  general,  current  products  of  European  aircraft  technology  are 
apparently  not  highly  dependent  on  the  products  of  American  research 
and  development  or  American  experience  (although  both  are  frequently 
asserted),  nor  are  there  any  striking  technological  differences  between 
the  products  of  European  aircraft  construction  practices  and  American 
aircraft  construction  practices  today.  Differences  exist  (and  in  some 
cases  they  may  be  extremely  important),  but  they  do  not  appear  to  be 

*It  is  perhaps  worth  note  that  European  technology  is  represented 
in  several  contemporary  U.S.  military  aircraft;  the  engine  and  heads 
up  display"  of  the  A-7,  the  wing  box  of  the  variable-sweep  F-14,  and 
the  Harriers  purchased  by  the  U.S.  Marines  are  well  known  examples. 


Table  5 


AIRCRAFT  DEVELOPMENT  EXPERIENCE 
(West  European  and  U.S.) 


Harrier 

(Kestrel) 

Jaguar 

(B.A.C.) 

Viggen 

Lightning 

(P-1B) 

Mirage 

IIIA 

Mirage 

IV 

F-111A 

Performance  (Mach) 

.9 

(VTO) 

1.6 

1.8 

2.2 

2.2 

2.2 

2.3 

Manning : 

Engineering  staff 

160  to  330 

320 

650 

400 

50 

70 

4000 

to 

Experimental  shop 

350  + 

300 

350 

300 

300 

400 

6000 

Total 

Program  office 
(Government) 

24 

35 

20 

20 

10 

12 

220 

Months  from  design 

start  to: 

First  flight 

22 

54 

43 

34 

16 

17 

25 

First  production 
item  delivered 

48 

64 

96a 

45 

38 

54 

58 

Number  of  test 
vehicles 

13 

6 

6 

5 

3 

4 

23 

Development 
cost  factor 

1.0 

1.1 

1.3 

1.4 

1.0 

1.1 

1.8 

3Includes  24-month  schedule  stretchout  to  reconcile  a  changed  threat  estimate  with  altered  budget 
constraints . 
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much  greater  than  the  differences  among  various  models  of  American 
combat  aircraft  of  the  1960s.  The  most  notable  difference  between 
typical  European  and  American  operational  aircraft  of  1970  is  in  the 
quantity  and  complexity  of  installed  avionics  equipment.  To  minimize 
the  influence  of  that  factor,  all  of  the  data  in  Table  5  concern  the 
basic  flight  vehicle  and  exclude  the  weapons-oriented  systems  for  each. 

The  last  horizontal  line  on  Table  5  lists  development  cost  factors 
of  the  sort  discussed  in  preceding  sections.  Two  points  need  to  be 
made  in  considering  them:  first,  actual  development  costs  are  not 
readily  obtainable  for  U.S.  aircraft  owing  to  our  accounting  practices, 
whereas  in  Europe  the  separation  of  production  from  development  tends 
to  make  development  costs  more  or  less  determinable:  second,  the  cost 
factors  for  aircraft  shown  in  Table  5  generally  exclude  avionics  devel¬ 
opment.  Development  cost  factors  for  D.S.  systems  (including  avionics) 
range  from  1.5  to  more  than  2.0.  For  the  F-111,  the  example  chosen, 
here,  the  cost  factor  for  airframe-engine  development  (excluding  avionics) 
is  at  least  1.8.  Inclusion  of  avionics  development  costs,  as  best  they 
can  be  estimated,  drives  the  development  cost  factor  to  at  least  2.0. 

In  recent  European  programs  examined  by  Rand,  the  typical  development 
cost  factor  appeared  to  be  about  1.2  or  1.3,  and  in  the  sample  examined 
there  was  no  development  cost  factor  larger  than  1.4.  Because 
definitive  information  on  the  absolute  cost  and  estimated  cost  of  recent 
European  aircraft  development  could  not  be  readily  obtained,  the  cost 
factors  for  European  aircraft  were  provided  by  government  offices  or 
contractor  organizations  who  used  Rand's  definitions  and  ground  rules. 

It  is  conceivable  that  the  numbers  are  biased,  but  cost  information 
independently  obtained  seems  to  corroborate  that  provided  by  the  Euro¬ 
peans.  Even  if  the  numbers  are  wrong  by  20  or  30  percent,  they  still 
are  revealing.  Perhaps  more  important,  the  costs  for  aircraft  develop- 
ment  evidently  are  substantially  lower  in  the  European  case. 

The  period  typically  required  to  proceed  from  design  start  to 
either  first  flight  or  to  the  availability  of  a  first  production  item 
is  not  significantly  longer  in  Europe  than  in  the  United  States.  Tne 
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Dassault  case,  which  has  been  widely  discussed  in  the  United  States 
recently,  is  truly  exceptional.  On  one  occasion,  Dassault  progressed 
from  design  start  to  the  first  flight  of  a  prototype  aircraft  in  nine 
months;  that  achievement,  which  occurred  in  the  late  1950s,  has  not 
been  bettered  in  the  United  States  since  1944,  when  the  P-80  went  from 
design  start  to  first  flight  in  six  months.  In  the  case  of  the  Mirage 
jy ,  the  development  program  was  marked  by  the  same  sort  of  uncertainty 
that  has  characterized  several  high-risk  American  programs.  For  example, 
at  the  time  design  began  Dassault  did  not  know  with  any  certainty  the 
required  size  of  the  bomber  or  the  weight  or  bulk  of  the  first  French 
nuclear  weapon.  Dimensional  assumptions  passed  to  Dassault  by  the 
French  government  ultimately  proved  to  be  invalid,  and  the  second 
engineering  test  aircraft  had  to  be  made  appreciably  larger  than  the 
first.  There  was  an  additional  complication  in  the  fact  that  the  engine 
preferred  by  Dassault  was  a  license-built  version  of  an  American  engine, 
and  the  French  Air  Ministry  was  extremely  reluctant  to  adopt  a  propul¬ 
sion  system  that  might  be  somehow  dependent  on  the  unpredictabilities 
of  American  international  policies.  Consequently,  a  French  engine  was 
installed  midway  through  the  development  program  and  the  desired  per¬ 
formance  was  adversely  affected. 

Aircraft  first  delivered  to  the  French  Air  Force  for  use  in  the 
Fovoe  de  FTCLppe  had,  therefore,  about  10  percent  less  performance  than 
planned.  Dassault  and  the  French  Air  Force  maintain  today  that  the 
performance  disparity  has  been  overcome  and  that  inventory  aircraft 
are  for  practical  purposes  comparable  in  performance  to  those  specified 
when  the  design  program  began.  Dassault  aircraft  at  least  the  Mirage 

series  —  are  evolutionary.  There  is  no  disputing  the  fact.  But  the 
70,000-pound  Mirage  IV  obviously  could  not  have  been  scaled  up  from 
a  Mirage  III,  which  weighed  less  than  25,000  pounds  fully  loaded,  with¬ 
out  considerable  design  ingenuity  and  originality. 

^Illustrated  in  Table  5  by  the  Mirage  III  and  Mirage  IV,  but  in¬ 
cluding  several  other  aircraft  in  the  Mirage  family. 
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Among  the  most  interesting  data  items  obtained  in  the  recent 
survey  of  European  aircraft  development  were  the  manning  numbers  for 
the  engineering  staff,  the  experimental  shop  where  engineering  test 
vehicles  were  constructed,  and  the  government  program  offices  or  their 
equivalents.  Fewer  engineers  were  employed  in  the  design  process  than 
in  comparable  American  design  staffs ,  in  some  cases  by  a  factor  of  ten 
and  in  almost  every  case  by  a  factor  of  three  or  more.  The  numbers  of 
supervisory  and  management  personnel  assigned  to  the  government  project 
office  appear  to  be  smaller  by  about  the  same  ratios.  The  technological 
and  performance  differences  that  distinguish  the  European  aircraft  in 
this  sample  from  comparable  American  aircraft  are  in  no  sense  large 
enough  to  explain  the  striking  differences  in  invested  manpower  resources. 

In  some  part  the  explanation  lies  in  the  fact  that  for  nearly  ten 
years  most  European  aircraft  developers,  and  their  governments,  have 
been  operating  under  severe  financial  constraints.  Also,  there  simply 
are  not  enough  aircraft  engineers  available  in  any  West  European  nation 
to  man  a  program  of  the  sort  that  is  ordinary  in  the  United  States. 

For  practical  purposes  the  Europeans  have  compensated  for  this  disparity 
in  available  resources  by  pruning  away  the  redundancies  of  the  develop¬ 
ment  process.  In  the  case  of  Dassault,  there  is  no  periodic  reporting 
of  progress,  no  milestone  reporting,  no  repeated  briefings  of  any  sort 
for  the  government.  Progress  accounting  is  on  an  irregular  basis  and 
by  letter.  In  the  case  of  the  British  (for  the  Harrier  project)  ,  pro¬ 
gress  is  reported  by  means  of  one  monthly  summary,  about  a  quarter  of 
an  inch  thick,  which  contains  synopses  of  all  important  events,  program 
status  evaluation,  and  program  projections.  The  contractor,  Hawker- 
Siddeley,  manages  the  Harrier  program  on  the  basis  of  a  comparable 
report  that  contains  about  12  or  15  additional  pages. 

In  Western  Europe,  very  few  engineering  personnel  are  engaged  in 
the  sort  of  data  analysis  that  is  common  to  American  programs.  The 
Swedes  indulge  rather  more  extensively  than  do  the  British  or  the 
French,  but  even  in  Sweden  the  amount  of  work  that  is  not  immediately 
relevant  to  the  task  at  hand  is  surprisingly  small.  True,  some  Ameri¬ 
can  management  devices  are  widely  used;  PERT,  for  example,  is  employed 
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by  both  SAAB  and  the  British  Aircraft  Corporation.  But  the  PERT  sche¬ 
matic  ordinarily  used  in  Europe  captures  no  more  than  6,000  to  10,000 
events .  A  short  and  uncluttered  technical  decision  net  is  characteris¬ 
tic.  No  more  than  three  or  four  approvals  are  required  —  all  at  the 
working  level  —  to  validate  drawings;  project  managers  have  direct 
access  to  the  two  or  three  people  whose  approval  is  necessary  to  fund 
design  changes  or  to  approve  them  in  terms  of  the  performance  implica¬ 
tions  for  the  system. 

The  fundamental  strategy  used  widely  in  Europe,  although  not  in 
all  programs,  of  course,  calls  for  no  substantial  production  commitment 
to  be  made  until  the  basic  development  process  has  been  completed  and 
proof  of  utility  demonstrated  by  performance  tests  of  an  engineering 
shop  article  that  is  reasonably  representative  of  the  desired  opera¬ 
tional  item.  The  common  terminology  is  "prototype,"  but  in  actual 
practice  what  is  done  is  strikingly  different  from  the  sort  of  activity 
usually  implied  when  the  term  "prototype"  is  used  in  the  United  States. 
In  particular,  engineering  shop  aircraft  are  ordinarily  assembled  on 
what  would  be  considered  "hard"  tooling  in  this  country  —  but  not  a 
full  set  of  hard  tooling.  The  Viggen  is  a  remarkable  case  in  point. 

To  complete  fabrication  of  six  engineering  shop  Viggen  aircraft,  SAAB 
spent  25  percent  of  the  total  tooling  budget  for  the  Viggen  program. 

At  the  end  of  the  engineering-shop  phase  of  development,  when  fabrica¬ 
tion  of  operational  aircraft  was  approved,  about  10  percent  of  the 
tooling  used  in  fabricating  the  first  six  aircraft  had  been  discarded 
or  replaced.  That  10  percent  throwaway  represented  only  about  two  and 
one-half  percent  of  the  total  tooling  budget  for  the  Viggen;  and  in 
that  instance  the  tooling  change  accommodated  a  relatively  substantial 
alteration  of  the  basic  fuselage  airframe  and  appreciable  changes  in 
both  the  lift  and  control  surfaces. 

In  lieu  of  full  hard  tooling,  Dassault,  SAAB,  BAC ,  and  Hawker- 
Siddeley  rely  on  highly  skilled,  permanent-cadre  engineering  shop 
personnel  to  assemble  the  experimental  shop  aircraft  to  the  same  sorts 
of  detailed  drawings  that  are  ordinarily  used  in  this  country,  but 
without  resort  to  full  sets  of  jigs  and  fixtures.  Indeed,  detailed 
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tooling  drawings  are  not  attempted  until  there  is  reasonable  assurance 
that  the  parts  to  be  fabricated  will  actually  be  used  in  operational 
aircraft . 

Another  element  of  the  design  process  in  Europe  that  is  extremely 
revealing  is  the  pronounced  reliance  on  early  proof  testing  of  engines, 
avionics,  and  airframes.  Production  commitments  are  delayed  until  both 
the  performance  and  the  probable  durability  of  the  subsystems  have  been 
appropriately  demonstrated. 

The  small  size  of  the  development  project  staff  is  an  important 
consideration  in  the  European  policy  of  proof  testing  systems  before 
committing  them  to  production.  A  lapse  of  several  months  between  com¬ 
pletion  of  a  test  item  and  commitment  to  production  makes  it  imprac¬ 
tical,  on  economic  grounds  alone,  to  maintain  a  large  project  staff 
while  testing  proceeds.  But  a  relatively  small  staff  of  highly  skilled 
designers  can  be  usefully  employed  at  many  tasks  and  in  extreme  cases 
can  be  “carried"  for  a  considerable  period  at  relatively  slight  cost. 

On  the  other  hand,  the  existence  of  a  large  design  staff  plus  production 
capability  common  in  this  country  represents  a  source  of  pressure  to 
move  at  once  to  the  next  stage  of  acquisition,  be  it  final  design  or 
production,  regardless  of  whether  the  current  development  or  test  phase 
has  been  satisfactorily  completed. 

The  delay  for  validation  of  performance  has  another  effect;  it 
permits  rational  and  low-cost  alterations  of  specifications  to  make 
actual  performance  correspond  realistically  to  a  reappraised  threat. 

The  Swedish  government  accepted  a  lower  level  of  performance  at  the 
mid-stage  of  development  of  the  Viggen  because  an  anticipated  threat 
had  not  materialized  —  and  seemed  unlikely  to  do  so  in  the  immediate 
future . 

In  general,  for  about  the  same  level  of  airframe  technology, 
European  schedules  tend  to  be  about  10  to  50  percent  longer  than  pre¬ 
dicted  U.S.  schedules  for  comparable  aircraft  systems,  but  only  10  to 
20  percent  longer  than  actual  U.S.  schedules.  The  difference  represents 
schedule  slips. 
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In  France  the  cost  of  military  aircraft  development  through  the 

flight  test  phase  may  be  but  one-fifth  and  rarely  is  as  much  as  one- 

* 

half  of  the  U.S.  cost  for  comparable  systems.  Nor  does  the  unit 
price  of  production  items  appear  to  be  appreciably  higher  than  for 
comparable  U.S.  aircraft.  Further,  and  perhaps  most  significant, 
recent  European  program  outcomes  have  tended  to  correspond  very 
closely  to  the  cost  predictions  made  when  system  development  began. 
Differences  between  anticipated  performance  and  actual  performance, 
when  they  occur,  generally  appear  to  have  been  chosen  deliberately  as 
alternatives  to  incurring  costs  higher  than  predicted.  But,  to  restate 
an  earlier  caveat,  European  indulgence  in  frequent  cost-performance 
tradeoffs  should  not  suggest  that  the  performance  of  European  aircraft 
is  appreciably  inferior  to  that  of  comparable  U.S.  aircraft.  In  most 
instances,  that  does  not  appear  to  be  the  case.  It  is  worth  recalling 
that  in  many  respects  the  performance  attributes  of  American  aircraft 
do  not  entirely  match  those  predicted  when  program  approval  was  granted. 
Nor  is  the  experience  of  the  Department  of  Defense  in  requiring  and 
obtaining  performance  superior  to  that  originally  projected  exclusive 
to  the  United  States;  improved  performance  is  also  observable  in  ad¬ 
vanced  models  of  most  European  fighter  aircraft  developments  although 
it  appears  to  be  introduced  more  systematically  and  somewhat  later  in 
the  development-production  cycle  than  in  U.S.  experience. 

THE  SOVIET  UNION 

Another  indication  that  strategy  differences  may  have  an  appre¬ 
ciable  effect  on  the  cost  and  other  outcomes  of  major  development 
programs  is  suggested  by  a  recent  survey  of  the  experience  of  the 
Soviet  Union  in  developing  aircraft  in  the  decades  since  the  close  of 
World  War  II.  Table  6  lists  the  numbers  of  bomber  and  fighter  aircraft 

Such  a  statement  should  not  be  interpreted  as  a  reflection  of 
an  elaborate  normalization  of  wage  rates,  materials  costs,  and  inten¬ 
sity  of  tooling.  We  have  not  completed  such  a  comparison. 

Obviously,  neither  the  Concorde  (Anglo-French  SST)  nor  the 
Rolls-Royce  RB-211  engine  fits  such  a  generalization.  But  they  were 
not  undertaken  in  accordance  with  customary  European  practices. 
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Table  6 

U.S. /SOVIET  AIRCRAFT  DEVELOPMENT  PROJECTS,  1945-1969 


United 

States 

Soviet  Union 

Total 

Entered  Service 

Total 

Entered  Service 

Fight  e 

rs  and  Attack 

Aircraft 

1945-49 

23 

14 

35 

11 

1950-54 

18 

17 

13 

5 

1955-59 

10 

8 

9 

4 

1960-64 

5 

4 

6 

4 

1965-69 

2 

2 

10 

5a 

Bombers 

1945-49 

15 

6 

10 

3 

1950-54 

8 

7 

8 

3 

1955-59 

3 

3 

9 

5 

1960-64 

2 

1 

3 

3 

1965-69 

1 

1 

1 

1 

aThree  known  to  have  entered  service;  two  additional  are 
"probable." 


A 
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that  the  United  States  and  the  Soviet  Union  have  carried  to  the  point  of 
publicly  observed  first  flight  since  1945.  It  will  be  noted  that  since 
1955,  a  period  that  basically  covers  development  of  all  aircraft  now  in 
service  and  those  likely  to  enter  service  in  the  next  few  years,  the 
Soviet  Union  has  carried  about  one  and  one-half  times  as  many  fighter 
aircraft  to  the  flight  test  stage  as  has  the  United  States,  and  about 
twice  as  many  bomber  aircraft. 

There  are  two  possible  explanations  for  this  contrast.  One  is 
that  the  cost  of  a  fighter  aircraft  development  is  approximately  the 
same  for  both  countries  and  that  the  Soviet  Union  is  spending  about 
twice  as  much  money  on  the  development  of  military  aircraft  as  is  the 
United  States  and  has  been  doing  so  for  about  ten  years.  Too  little 
is  known  about  the  resources  the  Soviets  have  invested  in  aircraft 
development  in  that  period  to  put  any  confidence  in  an  allegation  that 
such  is  or  is  not  the  case.  But  relatively  recent  studies  of  the 
Soviet  economy  and  the  Soviet  budget  make  it  seem  rather  unlikely  that 

* 

the  Soviet  Union  has  indeed  been  greatly  outspending  the  United  States . 
And  if  the  Soviet  Union  now  and  for  some  years  has  been  spending  no 
more  than  the  United  States  on  the  development  of  military  aircraft, 
one  is  driven  to  the  somewhat  alarming  conclusion  that  in  its  initial 
stages,  at  least,  the  Soviet  aircraft  development  process  is  appreciably 
less  costly  than  the  comparable  American  development  process.  It  is 
reasonably  clear  that  the  Soviets  develop  and  test  aircraft  in  a 
way  that  the  United  States  largely  abandoned  some  15  years  ago;  and  if 
evidence  of  West  European  experience  can  be  treated  as  applicable,  the 
strategy  favored  by  the  Soviet  Union  is  in  many  respects  less  costly 
than  our  own.  The  implication  is  that  the  Soviet  Union  actually  has 
created  at  equivalent  cost  more  aircraft  options  than  has  the  United 
States.  The  advantage  of  having  several  aircraft  available  for  selec¬ 
tion  (or  having  them  past  the  proof  test  stage)  and  of  being  able  at 

•(j 

"The  Soviet  Threat,"  a  summary  by  the  Chairman  of  the  Armed  Ser¬ 
vices  Committee,  House  of  Representatives,  suggests  (p .  9919)  that  the 
ratio  of  defense-related  R&D  spending  since  1955  has  been  about  6:5,  the 
U.S.  spending  more.  But  the  statement  notes  that  in  1969  and  1970,  the 
USSR  spent  somewhat  more. 
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that  point  to  choose  the  one  most  appropriate  to  the  imminent  threat, 
seems  clear.  The  available  evidence  on  Soviet  R&D  practices  is  far 

* 

from  conclusive;  further  research  in  this  area  seems  thoroughly  warranted. 


Studies  of  the  Soviet  design  process  and  its  characteristics 
have  been  in  progress  at  Rand,  on  a  small  scale,  for  several  months. 

The  initial  report  is  by  A.  J.  Alexander,  R&D  in  Soviet  Aviation:  The 
Relationship  Between  Organization  and  Outputs ,  R-589-PR,  November  1970. 

A  report  on  Soviet  science  policy  is  in  an  advanced  state  of  preparation. 
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IV.  TWO  PROPOSALS:  AUSTERE  DEVELOPMENT  AND 
AN  INCREMENTAL  ACQUISITION  STRATEGY  " 

The  analysis  of  36  major  DoD  systems,  together  with  observations 
of  European  aircraft  development  practices  and  non-standard  American 
programs,  identified  some  major  causes  of  high  system  cost.  The  list 
contains  no  surprises;  these  sources  of  cost  growth  have  been  singled 
out  in  past  studies  of  the  system  acquisition  process. 

High  system  cost  and  cost  growth  appear  to  arise  primarily  from 
efforts  to  subdue  difficult  technology  on  highly  compressed  schedules 
and  an  apparent  willingness  to  pay  whatever  is  required  to  insure 
satisfaction  of  original  (or  even  expanded)  system  performance  goals. 
Another  obvious  and  important  contributor  is  the  customary  acceptance 
of  optimistic  assumptions  about  the  long-term  predictability  of  tech¬ 
nology  and  the  cost  of  coping  with  it.  Occasionally  there  may  be  a 
valid  reason  for  urgency  in  satisfying  original  goals,  for  incorpo¬ 
rating  new  and  nx>re  stringent  requirements  during  a  program,  or  for 
insisting  upon  original  schedule  expectations.  There  is  little  evi¬ 
dence  that  extreme  urgency  characterizes  many  current  programs,  how¬ 
ever.  In  most  cases  the  appropriate  verdict  would  seem  to  be  "not 
proven." 

The  advocacy  of  more  efficient  and  more  economical  aircraft 
development  practices  has  been  a  tradition  in  this  country  for  at 
least  a  dozen  years.  Efforts  have  been  made  to  identify  and  eliminate 
the  causes  of  inefficiency,  but  such  efforts  have  ordinarily  been 
directed  at  refining  management  procedures  and  exercising  better  con¬ 
trol  over  the  activities  of  contractor  development  organizations. 
Attempts  to  impel  contractors  to  be  more  attentive  to  cost  considera¬ 
tions  generally  have  focused  on  some  variant  of  fixed-price  incentive 
contracting.  But  neither  greater  elegance  of  management  procedures 
nor  readiness  to  force  a  major  contractor  into  extreme  financial  dif¬ 
ficulties  has  alleviated  the  fundamental  problem.  The  success  of  some 
recent  development  and  acquisition  programs  that  have  been  conducted 
without  deference  to  most  of  the  usual  management  rituals  suggests 
that  a  strikingly  different  strategy  for  acquisition  may  be  appropriate. 
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The  evidence  reviewed  in  the  preceding  sections  demonstrates 
three  aspects  of  system  acquisition  that  are  too  thoroughly  supported 
to  be  classed  as  hypotheses  but  perhaps  not  sufficiently  documented 
to  qualify  as  truths . 

First,  despite  determined  efforts  to  improve  the  outcome  of  major 
acquisition  programs  by  altering  contractual  approaches  and  introducing 
complex  management  reforms,  recent  programs  have  incurred  cost,  schedule, 
and  system  performance  difficulties  not  greatly  different  from  those 
characteristic  of  the  1950s. 

Second,  although  some  advantage  could  be  gained  by  improving  the 
effectiveness  of  the  cost  estimating  process  and  by  applying  techniques 
that  would  encourage  the  earlier  identification  and  correction  of  the 
causes  of  cost  growth  in  individual  programs,  there  is  no  evidence  that 
substantial  improvements  in  the  outcomes  of  system  acquisition  programs 
can  be  anticipated  by  following  that  route. 

Third,  in  a  number  of  instances,  major  system  acquisition  programs 
that  departed  in  many  respects  from  the  pattern  ordinarily  imposed  by 
the  U.S.  Department  of  Defense  in  the  1960s  had  surprisingly  good  out¬ 
comes;  proved  remarkably  predictable  with  respect  to  cost,  performance, 
and  schedule;  and  were  appreciably  less  costly  in  terms  of  total  resources 
expended  than  comparable  programs  carried  through  by  the  more  ordinary 
techniques  of  DoD  system  acquisition. 

These  are  provocative  indicators  —  clearly,  there  is  room  for 
improvement  in  system  acquisition  policy  and  there  are  feasible  alter¬ 
natives.  In  all  candor,  we  must  point  out  that  such  evidence  as  is  now 
available  confirms  only  that  there  are  several  possible  alternatives 
that  can  potentially  improve  the  situation,  not  that  a  uniquely  superior 
remedy  exists.  Many  approaches  may  provide  relief,  but  based  on  our 
available  empirical  evidence,  the  following  strategy  seems  most  likely 
to  facilitate  substantial  improvements. 

*See  also,  for  example,  Blue  Ribbon  Defense  Panel,  Report  to  the 
President  and  the  Secretary  of  Defense  on  the  Department  of  Defense , 
Washington,  July  1970  (esp.  Appendix  E,  Staff  Report  on  Major  Weapon 
Systems  Acquisition  Process). 
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THE  PROPOSED  STRATEGY  CHANGES 

On  the  strength  of  evidence  derived  from  an  examination  of  indi¬ 
vidual  DoD  programs  exempted  from  ordinary  rules  of  system  acquisition, 
and  from  evidence  provided  by  examinations  of  current  and  recent 
European  aircraft  developments,  it  is  possible  to  construct  a  consistent 
set  of  principles  applicable  to  a  wide  variety  of  systems  that  may  be 
required  in  the  1970s. 

The  proposals  outlined  below  represent  a  sharp  departure  from  the 
policies  common  in  the  1960s  in  which  normally  a  single,  major  autho¬ 
rization  decision  was  made,  followed  by  a  highly  concurrent  program 
where  production  was  initiated  long  before  development  was  truly  com¬ 
pleted.  We  suggest  instead,  as  the  "normal"  approach  to  major  weapon 
system  acquisitions  for  the  1970s,  an  approach  characterized  by  two  key7 
principles:  (1)  an  incremental  strategy  involving  a  sequence  of  de¬ 

cision  points,  and  (2)  considerable  austerity  in  the  early  phases  of 
development . 

An  Incremental  Strategy 

The  evidence  gleaned  from  past  programs  indicates  there  is  much  to 
be  gained  from  conducting  acquisition  programs  in  discrete  phases 
clearly  separated  from  one  another.  Basically,  such  an  approach  would 
require  separating  the  development  of  systems  from  the  subsequent  pro¬ 
duction  of  those  systems;  furthermore,  it  would  call  for  first  con¬ 
ducting  those  aspects  of  development  aimed  at  demonstrating  the  per 
formance  potential  of  the  system  and  later  addressing  such  issues  as 
verifying  reliability  and  maintainability  of  the  system  and  providing 
for  the  special  constraints  imposed  by  service  support  requirements. 
Finally,  an  incremental  strategy  could,  and  ordinarily  would,  include 

It  is  assumed  here  that  the  design  phase  included  consideration 
of  reliability,  maintainability,  and  similar  factors;  such  is  clearly 
the  appropriate  and  usual  practice  in  most  aircraft  design  processes. 
Our  point  is,  however,  that  the  preliminary  design  phase  should  not 
include  an  elaborate  attempt  to  resolve  maintainability,  reliability, 
and  similar  issues  until  there  is  some  reasonable  assurance  that  the 
system  has  a  performance  relevant  to  need. 
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periodic  reassessment,  redefinition,  and  readjustment  of  such  program 

constituents  as  probable  cost,  performance  responses  to  a  changing 

threat,  schedule  objectives,  cost  implications  of  proposed  changes, 

and  residual  technical  advance  required  to  satisfy  program  objectives 

* 

at  any  given  stage  of  a  program. 

The  notion  of  an  incremental  acquisition  strategy  certainly  is 
not  new.  Some  variant  of  it  was  common  in  this  country  twenty  years 
ago,  and  for  practical  purposes  Europeans  have  been  using  it  through 
most  of  the  1960s.  In  many  instances  "exempt  programs"  have  used  a 
similar  strategy  recently  in  this  country,  and  it  is  the  ordinary 
way  in  which  many  commercial  aircraft  are  developed. 

Major  weapon  systems  can  be  divided  into  at  least  two  groups 
that  would  benefit  from  the  strategy  in  different  ways.  Suppose  the 
A-factor  concept  described  earlier  [see  Section  I,  especially  Figure 
2]  is  used  as  a  way  of  defining  the  groups  of  weapon  systems.  As 
observed  [Section  IX],  for  systems  with  A-factors  of  less  than  about 
12,  program  growth  is  reasonably  predictable.  For  programs  in  that 
group,  the  major  benefit  of  an  incremental  acquisition  strategy  would 
be  potentially  lower  system  costs,  arising  from  the  possibility  of 
maintaining  competition  to  the  time  of  production  decision  —  either 
by  conducting  two  or  more  development  programs,  or  through  competi- 
tive  initial  or  follow-on  production.  For  this  group  of  systems, 

it  seems  unnecessary  to  schedule  major  decision  points  during  the 
development  phase;  a  relatively  low  A-factor  value  implies  that  the 
performance  outcome  is  reasonably  predictable.  The  Air  Force’s  A-X 
program,  which  appears  to  fall  in  this  group,  is  perhaps  the  best 
current  example  of  applying  such  a  strategy. 

The  technique  involves  a  tradeoff  evaluation  that  balances  the 
cost  of  obtaining  whatever  performance  has  not  yet  been  demonstrated 
against  the  urgency  of  incorporating  still-to-be  achieved  performance 
in  the  article  finally  produced. 

Lack  of  "austerity"  in  two  or  more  development  programs  can 
easily  reduce  and  possibly  eliminate  the  benefits  of  competition  in 
choosing  the  production  contractor;  see  p.  48. 

***On  some  other  aspects  of  the  question  of  separating  production 
from  development,  see  below,  pp.  45-46. 
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The  most  potentially  significant  application  of  an  incremental 
acquisition  strategy  may  be  for  weapon  system  developments  having 
A-factor  values  greater  than  12  —  where  the  current  acquisition 
strategy  has  generally  been  unsuccessful.  In  these  cases,  tradeoffs 
between  cost  and  performance,  an  intentional  extension  of  a  develop¬ 
ment  program  to  provide  more  time  for  resolving  unforeseen  difficulties, 
or  a  restructuring  of  objectives  to  counter  unanticipated  threats  may 
be  vital  to  controlling  program  outcomes.  It  might  even  be  desirable 
to  terminate  a  development  program  if,  for  example,  the  technology 

sought  were  not  attainable  at  a  reasonable  price  or  the  threat  failed 
* 

to  materialize.  The  periodic  reassessment  implied  by  application  of 
an  incremental  strategy  suggests  recurrent  evaluation  not  only  in 
terms  of  its  intrinsic  promise,  but  also  of  its  advantages  over  com¬ 
peting  systems  also  in  development  or  already  in  the  force.  There 
has  not  been  a  recent  example  of  such  an  approach,  but  the  strategies 
used  in  the  period  of  early  ICBM  development  (leading,  ultimately, 
to  the  Atlas  and  Titan)  generally  resemble  those  applicable  to  such 
high  technology  programs. 

The  characteristic  response  to  difficulties  encountered  in  "con¬ 
current”  programs  of  the  past  has  been  based  on  the  assumption  that  more 
careful  advance  planning,  more  thorough  analysis,  and  more  extensive  pre¬ 
program  design  work  would  compensate  for  the  uncertainties  that  troubled 
earlier  programs.  Technical  uncertainty  and  the  development  process 
have  been  inseparable  companions  since  long  before  aircraft  and  missile 
development  began.  In  any  program  marked  by  a  considerable  degree  of 
technical  uncertainty,  it  is  extremely  unlikely  that  planners  can  anti¬ 
cipate  the  precise  nature  of  the  difficulties  and  take  steps  in  advance 
to  resolve  them.  Careful  analysis  and  planning  in  advance  of  program 
approval  should  not  be  abandoned,  of  course.  But  planners  should  not 
delude  themselves  into  believing  that  abstract  planning,  however  com¬ 
prehensive,  will  resolve  technical,  scheduling,  or  cost  uncertainties, 

* 

The  difficulties  of  evaluating  a  SPO  manager  under  such  condi¬ 
tions  are  not  to  be  minimized.  A  very  successful  development-only 
program  could  occur  in  which  the  potential  defensive  system  induced 
prospective  adversaries  not  to  deploy  a  new  offensive  system. 
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and  that  programs  will  thereafter  proceed  toward  completion  without 
encountering  difficulty  and  without  being  subject  to  change  in  both 
general  goals  and  small  details. 

In  the  course  of  development  of  a  new  weapon  system  it  may  become 
apparent  that  the  performance  goal  need  not  be  precisely  that  origi¬ 
nally  specified  or,  alternatively,  that  the  performance  originally 
specified  can  be  attained  only  at  a  cost  much  greater  than  originally 
proposed.  The  ordinary  response  to  that  circumstance  in  the  1960s 
was  to  adhere  to  original  performance  requirements  or  to  incorporate 
requirements  for  improved  performance  and  to  accept  the  cost  conse¬ 
quences  of  either  action.  In  some  cases  the  effort  to  incorporate 
performance  substantially  beyond  the  state  of  the  art  of  the  time  has 
caused  a  great  deal  of  money  to  be  expended  without  ultimately  pro¬ 
viding  the  desired  performance.  The  maximum  speed  of  the  F-106,  the 
range  of  the  B-58,  and  the  supersonic  range  capability  of  the  F-lll 
are  relatively  recent  instances  of  originally  specified  performance 
that  could  not  be  attained  at  an  acceptable  cost.  The  structural 
weight  requirement  for  the  C-5A  represents  performance  that  was  attained 
but  probably  was  not  worth  the  additional  cost.  When  the  discovery 
of  probable  performance  shortfalls  or  recognition  of  the  need  for 
additional  performance  features  occurs  at  an  early  stage  of  development 
a  fresh  examination  of  other  alternatives  is  appropriate.  For  example, 
if  some  aspect  of  the  originally  required  performance  (or  a  more  demand¬ 
ing  performance  requirement)  is  attainable  only  with  the  expenditure 
of  a  great  deal  of  additional  money,  it  might  be  highly  desirable  to 
choose  among  the  available  alternatives  rather  than  simply  to  accept 
the  costs  of  meeting  the  original  or  enlarged  performance  objectives. 

Similarly,  should  a  postulated  threat  change  character  before  pro¬ 
duction  commitments  have  been  made,  it  would  be  reasonable  to  change  an 
aircraft  or  missile  system  specification  to  compensate  for  the  condi¬ 
tions  of  the  changed  threat.  Such  a  change  is  less  feasible  from  a 
design  standpoint  —  and  more  expensive  —  if  production  has  begun  or 
if  all  the  costly  arrangements  for  production  have  been  made.  Again, 
should  difficulty  occur  in  development,  it  might  be  advisable  to  stretch 
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out  the  development  schedule  and  incur  the  modest  additional  cost  of 
resolving  that  difficulty  instead  of  simply  allocating  vastly  greater 
resources  to  the  program  (Buch  as  those  involved  in  extensive  retool¬ 
ing)  in  an  attempt  to  resolve  technical  difficulty  while  maintaining 
the  original  schedule. 

Finally,  let  us  consider  two  premises  often  accepted  by  develop¬ 
ment  agencies  throughout  the  DoD:  first,  that  the  development  and 
production  aspects  of  acquisition  must  overlap,  and  second,  that 
competition  for  production  aspects  of  the  acquisition  phase  cannot  be 
conducted  independently  of  competition  for  the  development  phases. 

Implicit  in  the  assertion  that  development  and  production  must 
overlap  is  the  assumption  that  in  an  incremental,  sequential  process 
there  is  an  unconscionably  long  wait  between  the  start  of  a  program 
and  the  delivery  of  an  operationally  ready  system.  Rand  studies 
conducted  several  years  ago  suggest  that  the  time  required  for  the 
development  of  a  system  by  incremental  processes  is  no  greater  than 
the  time  required  for  the  development  of  a  similar  system  by  "con¬ 
current"  processes.*  If  that  is  the  case,  and  the  evidence  is  no 
stronger  for  one  approach  than  the  other,  the  "normal"  strategy  for 
system  acquisition  in  the  1970s  should  involve  a  conscious  decision 
to  produce  (or  not  to  produce)  only  after  the  development  is  completed. 
However,  setting  down  such  a  procedure  does  not  guarantee  compliance 
--  the  proposed  strategy  is  by  no  means  foolproof.  If  the  first  part 
of  the  "production  phase"  is  used  to  incorporate  additional  "easy" 
development  improvements,  the  result  could  be  a  quick  return  to  current 
practices . 

The  second  premise,  concerning  the  assumed  infeasibility  of  separa¬ 
ting  competition  for  production  from  competition  in  the  development 

*See  B.  H.  Klein,  T.  K.  Glennan,  Jr.,  and  G.  H.  Shubert,  The  Role 
of  Prototypes  in  Development 3  The  Rand  Corporation,  RM-3467-1-PR,  April  ^ 
1971.  We  should  point  out  that  under  either  "concurrent"  or  "incremental 
procedures,  essential  long-lead- time  tooling  will  have  to  be  ordered 
during  the  development  phase.  The  incremental  strategy,  however,  avoids 
the  cost  of  full  production  tooling  before  the  system  has  been  fully 
developed . 
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phase,  is  not  supported  by  history.  Experience  with  aircraft  built 
under  license  by  industry  groups  functionally  and  geographically  inde¬ 
pendent  of  the  development  group  suggests  that  technology  transfer  is 

* 

in  many  instances  quite  feasible  —  if  not  always  easy.  In  particular, 
it  may  be  recalled  that  quite  recently  such  complex  aircraft  systems 
as  the  F-104G,  F-104J,  T-33,  F-4,  and  a  surprisingly  wide  variety  of 
European  aircraft  have  been  built  under  license  in  countries  other 
than  those  in  which  they  were  developed.  In  the  1950s,  the  B-47  air¬ 
craft  was  produced  by  both  Douglas  and  Lockheed,  as  well  as  by  its 
original  developer,  Boeing.  The  B-57  aircraft  built  in  this  country 
by  Martin  was  a  modified  version  of  the  Canberra  light  bomber  developed 
by  the  English  Electric  Corporation  in  the  late  1940s.  Ships,  aircraft, 
missiles,  torpedos,  tanks,  and  a  large  variety  of  avionics  equipment 
have  been  similarly  produced  in  plants  quite  distant  and  separate  from 
those  of  the  original  developer.  The  same  is  true  of  a  great  many  com¬ 
plex  articles  sold  commercially.  Although  technology  transfer  has  not 
been  uniformly  successful,  there  are  few  striking  examples  of  extreme 
failures  when  production  has  been  separated  from  development;  conversely, 
several  such  programs  have  had  very  successful  outcomes.  Technology 
transfer  makes  separation  of  development  and  production  feasible.  If 
the  sponsoring  agency  has  all  rights  to  the  designs  and  data  generated 
by  a  development  program,  competition  in  the  production  phase  becomes 
feasible.  The  advisability  of  separating  development  from  production 
depends  on  circumstances;  its  appropriateness  would  have  to  be  deter¬ 
mined  on  a  program-by-program  basis.  But  to  disregard  the  option 
seems  singularly  shortsighted. 

Austere  Initial  Development 

Conducting  an  acquisition  program  in  discrete  phases  with  pauses 
deliberately  and  systematically  introduced  between  the  phases  would 

*See  also  G.  R.  Hall  and  R.  E.  Johnson,  Aircraft  Co-Production 
and  Procurement  Strategy 3  The  Rand  Corporation,  R-450-PR,  May  1967. 

Including,  among  many,  the  Gnat,  Magister,  Mirage  IIIS,  Fiat 
G-90 ,  and  MiG-21. 
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encourage  the  sort  of  reassessment  necessary  to  the  tradeoff  process 
sketched  above.  To  be  of  maximum  effectiveness,  such  a  policy  must 
include  an  additional  objective  —  program  resources  (particularly 
manpower)  must  be  constrained.  Large  programs  employing  many 
thousands  of  people  resist  change  and  are  difficult  and  costly  to  slow 
or  redirect,  however  desirable  the  change  or  redirection  may  be. 

It  is  particularly  important  that  program  resources  be  constrained 
early  in  the  development  phase,  because  it  is  there  that  most  of  the  im¬ 
portant  changes  resulting  from  "discoveries"  are  made.  The  notion  in¬ 
troduced  earlier,  separating  the  development  phase  aimed  at  proving 
performance  from  subsequent  phases  aimed  at  making  the  system  reliable 
and  operationally  suitable,  suggests  that  resources  should  be  sharply 
constrained  during  the  "performance  demonstration"  phase.  After  the 
system  design  has  demonstrated  satisfactory  performance,  and  the  need 
for  a  system  of  that  type  has  been  reverified,  it  should  be  possible 
to  proceed  to  the  subsequent  phases  with  considerable  confidence  that 
most  of  the  major  changes  (and  surprises)  have  already  appeared.  During 
development,  information  is  the  desired  product,  and  one  should  attempt 
to  purchase  only  information  relevant  to  the  problem  being  addressed 
if  the  goal  is  development  efficiency. 

Another  way  of  stating  the  basic  principle  is  that  there  are  signi¬ 
ficant  cost  advantages  to  delaying  work  not  relevant  to  the  phase  of 
development  in  progress.  For  example,  it  is  unlikely  in  the  extreme 
that  spares  consumption  estimates  made  before  the  completion  of  an  air¬ 
craft  design  will  have  much  relevance  to  the  requirements  of  that  air¬ 
craft  when  it  is  introduced  into  service  four  to  eight  years  later.  On 
the  other  hand,  spares  and  consumption  rates  and  maintenance  requirements 
could  be  calculated  with  fair  accuracy  once  test  articles  were  in  hand 
for  evaluation  and  system  test  time  had  been  accumulated.  Plainly,  the 
designers  of  the  initial  system  should  pay  attention  to  reliability  and 
maintenance  considerations;  such  considerations  play  a  vital  role  in  the 
initial  design  phase.  However,  the  extensive  and  expensive  work  necessary 
to  verify  system  reliability  and  to  design  a  maintenance  and  spare  parts 
stockage  policy  can  and  should  be  deferred  until  it  is  clear  what  the 
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system  configuration  will  be  and  that  the  system  will  in  fact  be  put 
into  production  and  operation. 

The  feasibility  of  conducting  successful  development  programs  under 
resource  constraints  of  this  sort  has  been  amply  demonstrated  in  the  past, 
and  a  few  notable  examples  were  mentioned  in  Section  III.  There  are  two 
classes  of  benefits  that  might  accrue.  One,  noted  above,  is  that  the 
direction  and  goals  of  small,  austere  programs  can  be  changed  quickly 
and  cheaply  in  response  to  technical  problems  or  shifting  requirements. 
There  are  numerous  examples  in  the  recent  past  of  large  development 
programs  wherein  major  difficulties  and  deficiencies  were  uncovered 
quite  early,  but  with  a  staff  of  several  thousand  people  working  on 
the  project  it  was  virtually  impossible  to  delay  or  redirect  the  pro¬ 
gram  to  accommodate  those  problems.  Another  potential  benefit  of 
initial  austerity  is  that  it  makes  possible  the  funding  of  multiple, 
competitive  sources  during  the  early  development  phases.  For  example, 
the  total  development  cost  of  a  modern,  high-performance  fighter  air¬ 
plane  today  is  in  the  neighborhood  of  $2  billion.  The  "preliminary 
development  cost"  of  a  conventional  program,  through  flight  test, 
approximates  $250  million.  However,  a  "performance  demonstrator" 
could  be  designed,  built,  and  flight  tested  for  less  than  $75  million.  * 
Hence,  for  an  additional  cost  of  well  under  5  percent  of  the  total 
development  cost,  it  seems  possible  to  carry  a  second,  competitive 
development  through  the  initial  hardware  phase. 

Discussions  of  a  low-cost  performance-demonstration  phase  frequently 
elicit  visions  of  an  "experimental"  system,  something  "built  on  hobby- 
shop  tooling"  that  has  little  direct  relation  to  the  item  subsequently 
put  into  quantity  production.  That  is  definitely  not  the  approach  being 
suggested  here.  Experience  here  and  abroad  demonstrates  that  by  defer¬ 
ring  expenses  related  only  to  quantity  production  or  operational  support 
(rate  tooling,  spares  provisioning,  extensive  reliability  testing,  and 

* 

Such  an  estimate  is  predicated  not  solely  on  recent  European  ex¬ 
perience,  but  also  on  informal  proposals  from  major  U.S.  constructors, 
unpublished  calculations  made  at  Rand  several  years  ago,  and  the  outcomes 
of  a  few  contractor-funded  programs  of  the  1960s. 
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so  on)  it  is  possible  to  build  initial  test  items  that  are  in  major 
respects  identical  to  the  version  then  postulated  for  production.  For 
example,  in  the  Mirage  and  Viggen  aircraft  programs  discussed  in  Section 
III,  the  initial  development  test  items  were  structurally  and  configura¬ 
tionally  identical  to  the  intended  operational  versions;  indeed,  the 
tooling  used  in  the  fabrication  of  the  initial  items  was  subsequently 
used  on  the  production  aircraft. 

EXPECTED  CONSEQUENCES  AND  BENEFITS 

The  potential  advantages  of  the  strategy  proposed  by  this  study 
are  substantial.  First,  an  incremental  acquisition  strategy  in  com¬ 
bination  with  a  policy  of  austerity  of  resource  expenditure  during  the 
early  phases  of  development  would  definitely  permit  more  new  program 
starts  at  a  cost  equal  to  or  even  less  than  that  incurred  in  the  ac¬ 
quisition  process  at  the  present  time.  Indeed,  even  though  it  does 
not  seem  entirely  possible  to  apply  to  American  programs  all  of  the 
practices  favored  abroad  (for  example,  by  the  French) ,  there  are  clear 
indications  that  major  aircraft  developments  could  be  carried  through 
the  initial  flight-test  stage  for  amounts  totaling  no  more  than  one- 
half  and  perhaps  as  little  as  one-fourth  of  the  amounts  now  required 
to  carry  programs  of  a  similar  nature  to  that  stage.  If  one  accepts 
such  a  premise,  it  would  be  entirely  feasible  to  carry  two  modestly 
different  fighter  aircraft  through  the  initial  development  phase,  and 
at  the  end  of  that  phase  to  consider  not  merely  which  was  the  better 
of  the  two  from  a  cost  and  performance  standpoint,  but  which  was  better 
suited  to  the  threat  then  anticipated.  Nor  is  it  inconceivable  that 
the  government  could  adopt  a  policy  of  purchasing  all  rights  to  products 
offered  in  such  a  competition  and  thus  could  integrate  the  better  ele¬ 
ments  of  two  slightly  different  systems  into  one  superior  system  through 
a  further  development  phase.  There  would  also  be  the  option  to  make 
the  production  selection  competitively  and  thus  carry  the  buyer's  bar¬ 
gaining  leverage  much  further  into  programs  than  is  possible  under 

* 

See  the  more  complete  discussion  of  this  point  in  Section  III, 

p.  33. 
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present  circumstances.  The  feasibility  of  technology  transfer,  which 
could  allow  competing  for  production  as  well  as  for  design  and  develop¬ 
ment,  has  been  previously  discussed.  The  complications  and  possible 
shortcomings  of  such  an  approach  clearly  require  a  more  thorough 
examination,  but  the  evidence  now  available  does  not  warrant  a  nega¬ 
tive  prejudgment  of  the  feasibility  of  such  an  option. 

Another  obvious  advantage  of  an  incremental  approach  is  that  the 
cost  of  production  articles  could  be  more  accurately  estimated  once 
an  engineering  test  article  reasonably  representative  of  the  desired 
final  product  becomes  available.  The  feasibility  of  introducing 
specification  changes  at  that  stage  should  also  be  much  clearer.  And 
of  course,  introducing  specification  changes  at  the  time  production 
begins  is  appreciably  less  expensive  than  introducing  such  changes  into 
ongoing  production  programs.  Indeed,  it  is  only  prudent  to  consider 
very  carefully  the  advantages  and  disadvantages  of  introducing  production- 
phase  specification  changes  as  block  changes,  and  deliberately  avoiding 
the  notoriously  expensive  process  of  retrofitting. 

Finally,  because  the  predictability  of  a  program  outcome  is  improved 
by  application  of  an  incremental  acquisition  strategy,  because  costs 
for  the  development  phase  tend  to  be  lower  and  more  controllable,  and 
because  there  is  a  higher  probability  that  the  system  will  be  responsive 
to  a  threat  that  exists  at  the  time  the  system  is  delivered,  the  credi¬ 
bility  of  the  service  proposing  production  of  that  system  and  postulating 
delivery  costs  should  be  appreciably  better  than  is  now  the  case.  One 
of  the  objections  to  the  current  system  of  acquiring  major  weapons  is 
that  program  outcomes  are  highly  unpredictable.  A  demonstration, 
through  one  or  two  successful  programs,  that  the  services  actually 
can  anticipate  program  outcomes  with  reasonable  precision  should  do 
a  great  deal  to  reestablish  the  credibility  of  the  individual  services. 

And  that  would  be  no  small  achievement. 

One  American  contractor  who  has  quite  a  lot  of  experience  with 
retrofitting  and  with  the  introduction  of  significant  changes  in  the 
production  line  estimated  that  the  former  process  is  about  four  times 
as  expensive  as  the  latter. 
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Some  of  the  more  commonly  stated  objections  to  an  incremental 
acquisition  strategy  deserve  consideration.  For  example,  an  incre¬ 
mental  program,  with  deliberate  pauses  to  evaluate  the  course  of  the 
program  and  its  future  prospects,  might  allow  greater  opportunity  for 
program  cancellation.  For  practical  purposes,  this  is  not  a  relevant 
objection.  "Bad"  systems  have  on  occasion  been  hurried  through 
development;  they  frequently  are  unsatisfactory  in  service  use  and 
tend  to  be  phased  out  earlier  than  had  originally  been  planned.  Nor 
is  development  by  a  concurrent,  compressed  process  any  guarantee  of 
program  completion.  In  a  development  situation  that  encourages  the 
start  of  several  low-cost  systems,  the  opportunity  for  carrying  one 
of  those  through  to  operational  delivery  is  appreciably  higher  than 
is  the  probability  of  completing  a  single,  not  very  promising  program. 

Alternatively,  it  is  often  assumed  that  an  incremental  strategy 
incorporating  the  elements  of  sequential  development  will  introduce 
unnecessary  steps  in  the  acquisition  process.  But  whatever  steps  are 
necessary  to  the  eventual  development  of  an  operationally  useful  system 
must  be  taken  in  any  system  development  process.  To  approach  them 
sequentially  and  to  resolve  one  difficulty  before  taking  on  a  second 
that  arises  from  the  first  is  not  an  exotic  technique. 

The  proposition  that  making  explicit  cost-versus-performance  com¬ 
promises  would  cause  the  eventual  delivery  of  systems  that  do  not  satisfy 
the  original  specifications  does  not  seem  relevant  either.  First,  as 
suggested  by  the  examination  of  experience  of  the  1960s,  systems  are 
being  delivered  today  that  in  many  respects  do  not  satisfy  original 
performance  requirements.  Second,  as  noted  above,  in  many  cases  de¬ 
sired  performance  is  never  achieved  even  if  a  great  deal  of  money  is 
spent  in  seeking  it.  Third,  in  some  cases  the  performance  that  is 
specified  for  a  system  is  not  always  essential  to  the  eventual  operational 
use  of  the  system.  The  threat  may  no  longer  be  relevant,  the  operating 
conditions  that  originally  suggested  development  of  the  system  may  have 
changed,  or  (as  is  frequently  the  case)  a  system  obtained  nominally  for 
one  military  mission  is  used  in  quite  another  assignment. 
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An  admission  of  uncertainty  at  the  start  of  a  program  —  and  the 
adoption  of  a  program  strategy  requiring  periodic  reevaluation  of  pro¬ 
gram  objectives,  costs,  and  schedules  —  will  not  necessarily  suggest 
to  DoD  or  Congressional  authorities  that  such  programs  may  not  be  car¬ 
ried  to  a  successful  conclusion.  At  present,  it  is  difficult  to  sell 
almost  any  program  to  a  DoD  and  a  Congress  that  have  little  confidence 
in  the  program-outcome  predictions  made  by  the  services.  The  product 
of  an  incremental  acquisition  strategy  would  in  many  instances  be  the 
resolution  of  uncertainties  at  an  early  stage  in  the  acquisition 
process  and  at  relatively  low  cost;  thereafter,  the  services  should 
be  in  a  reasonable  position  to  make  a  program  projection  that  would 
be  credible  to  the  DoD  and  Congress. 

IMPLICIT  ORGANIZATIONAL  RESTRUCTURING 

The  complexity  of  individual  weapon  systems  has  been  steadily  in¬ 
creasing.  Partly  in  consequence,  the  cost  of  development  and  the  unit 
cost  of  production  items  have  increased  greatly  in  recent  years.  Owing 
in  part  to  the  growing  cost  of  major  systems  and  in  part  to  budgetary 
pressures  that  reduce  the  amount  of  funds  available  for  the  development 
and  purchase  of  new  systems,  quantities  purchased  for  the  inventory 
have  steadily  decreased.  Furthermore,  the  high  and  poorly  predictable 
cost  of  developing  and  acquiring  systems  and  the  resultant  decreased 
credibility  of  the  advocating  services  have  made  it  increasingly  dif¬ 
ficult  for  the  services  to  obtain  approval  to  begin  new  system  develop¬ 
ments.  As  a  result,  in  the  last  decade  new  system  starts  have  been  less 
frequent,  and  planners  therefore  have  felt  obliged  to  attempt  larger 
technological  advances  for  new  systems  that  enter  development.  A 
secondary  but  perhaps  inescapable  consequence  of  those  circumstances 
has  been  the  increasing  difficulty  of  carrying  systems  through  develop¬ 
ment  successfully  and  expeditiously;  both  operating  commands  and 
developing  agencies  have  found  new  systems  troublesome  during  the 
period  of  their  introduction  into  regular  service. 

It  is  clear  that  the  adoption  of  an  incremental  acquisition  strategy 
and  a  policy  of  austerity  for  the  initial  stages  of  development  would 
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cause  some  major  changes  in  existing  institutions  and  policies,  but 
such  an  approach  has  the  potential  of  disrupting  the  unhappy  cycle  of 
events,  noted  above,  that  characterized  the  previous  decade.  Austere 
development,  for  example,  requires  smaller  design  groups,  smaller 
development  staffs,  and  thus  obviously  less  expensive  test  phases  than 
are  customary  at  the  present  time.  The  size  of  the  average  industry 
group  concerned  with  those  phases  of  development  would  in  all  prob¬ 
ability  decrease  very  substantially.  Additionally,  were  a  policy  of 
limited  reporting  and  direct  decision  routes  adopted,  the  size  of  the 
government  institutions  now  concerned  with  development  should  decrease 
proportionately .  Given  the  institutional  pressures  that  exist,  it  may 
not  be  possible  to  reduce  the  size  of  government-monitoring  establish¬ 
ments  to  those  ordinarily  encountered  in  West  European  countries  or 
in  such  "exempt"  programs  as  the  Agena  D.  Nevertheless,  project  groups 
and  decision-monitoring  groups  would,  in  all  probability,  employ  fewer 
than  one-half  and  perhaps  as  little  as  one-third  as  many  people  as  are 
employed  by  the  establishments  now  charged  with  acquiring  major  systems. 

Securing  an  early  commitment  from  senior  officials  of  the  DoD  and 
from  Congressional  authorities  is  essential  for  a  major  program  start. 

A  policy  of  refusing  to  make  early  commitments  to  major  programs  and 
delaying  decisions  on  production  and  deployment  until  there  is  sub¬ 
stantial  evidence  that  the  system  can  be  built  for  the  assumed  cost 
and  can  actually  respond  to  a  relevant  threat  would  represent  a  signifi¬ 
cant  departure  from  past  policies.  Changes  that  tend  to  reduce  the 
size  and  therefore  the  cost  of  the  present  development  establishment 
may  well  be  inescapable,  given  current  national  budget  trends.  From 
the  standpoint  of  the  services,  there  is  a  good  deal  to  be  said  for 
choosing  a  new  acquisition  strategy  rather  than  having  it  imposed  by 
the  upper  echelons  of  DoD  or  by  Congressional  authorities  impatient 
with  refusal  of  the  services  to  respond  appropriately  to  changed  bud¬ 
getary  circumstances. 

SUMMARY 

The  adoption  of  a  strategy  of  incremental  acquisition  and  a  policy 
of  austere  development  seems  almost  certain  to  result  in  lessened  cost 
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growth  and  lower  real  cost.  The  predictability  of  schedule  and  per¬ 
formance  outcomes  seems  almost  equally  certain  to  improve.  That  more 
system  options  could  be  generated  at  no  increase  in  present  system 
acquisition  cost  and  that  the  options  would  be  more  responsive  to 
emerging  threats  seem  evident.  Finally,  systems  would  probably  be 
delivered  with  fewer  performance  shortcomings  than  in  recent  years. 

The  evidence  is  not  all  in,  nor  does  it  make  an  absolutely  clear 
case  for  the  adoption  of  an  incremental  acquisition  strategy.  Further, 
an  incremental  acquisition  strategy  would  not  necessarily  be  equally 
and  uncritically  applicable  to  all  of  the  systems  proposed  for  develop¬ 
ment  in  the  next  decade.  Quite  obviously,  threats  may  arise  that  will 
demand  exemptions  from  the  proposed  strategy  and  acceptance  of  the 
risk  of  uncertain  program  outcomes  and  of  proceeding  toward  major 
systems  by  processes  approximating  "concurrency."  But  even  a  cursory 
examination  of  experience  in  the  last  six  or  eight  years  suggests  very 
strongly  that  few  programs  are  driven  to  fruition  by  such  a  dominant 
threat . 

For  many  if  not  most  new  systems,  the  attractions  of  an  incremental 
acquisition  strategy  and  austere  development  seem  persuasive.  Such 
evidence  as  is  available  supports  the  conclusions  that  a  major  change 
in  the  basic  strategy  of  acquiring  weapon  systems  is  desirable  and 
that  austere  development  and  an  incremental  acquisition  strategy  are 
very  attractive  options. 
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